Original: C ORRECTING THE S UB - OPTIMAL B IT A LLOCATION
Revised: C ORRECTING THE S UB - OPTIMAL B IT A LLOCATION

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: In this paper, we investigate the problem of bit allocation in Neural Video Compression (NVC).
Revised: In this paper, we investigate the problem of bit allocation in Neural Video Compression (NVC).

Original: First, we reveal that a recent bit allocation approach claimed to be optimal is, in fact, sub-optimal due to its implementation.
Revised: First, we reveal that a recent bit allocation approach claimed to be optimal is, in fact, sub-optimal due to its implementation.

Original: Specifically, we find that its sub-optimality lies in the improper application of semi-amortized variational inference (SAVI) on latent with non-factorized variational posterior.
Revised: Specifically, we find that its sub-optimality lies in the improper application of semi-amortized variational inference (SAVI) on latent with non-factorized variational posterior.

Original: Then, we show that the corrected version of SAVI on non-factorized latent requires recursively applying back-propagating through gradient ascent, based on which we derive the corrected optimal bit allocation algorithm.
Revised: Then, we show that the corrected version of SAVI on non-factorized latent requires recursively applying back-propagating through gradient ascent, based on which we derive the corrected optimal bit allocation algorithm.

Original: Due to the computational in-feasibility of the corrected bit allocation, we design an efficient approximation to make it practical.
Revised: Due to the computational in-feasibility of the corrected bit allocation, we design an efficient approximation to make it tractable.

Original: Empirical results show that our proposed correction significantly improves the incorrect bit allocation in terms of R-D performance and bitrate error, and outperforms all other bit allocation methods by a large margin.
Revised: Empirical results show that our proposed correction significantly improves the incorrect bit allocation in terms of R-D performance and bitrate error, and outperforms all other bit allocation methods by a large margin.

Original: The source code is provided in the supplementary material.
Revised: The source code is provided in the supplementary material.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Recently, bit allocation for Neural Video Compression (NVC) has drawn growing attention thanks to its great potential in boosting compression performance.
Revised: Recently, bit allocation for Neural Video Compression (NVC) has drawn growing attention thanks to its great potential in boosting compression performance.

Original: Due to the frame reference structure in video coding, it is sub-optimal to use the same R-D (Rate-Distortion) trade-off parameter λ for all frames.
Revised: Due to the frame reference structure in video coding, it is sub-optimal to use the same R-D (Rate-Distortion) trade-off parameter λ for all frames.

Original: In bit allocation task, bitrate is allocated to different frames/regions to minimize R-D cost R + λD , where R is total bitrate, D is total distortion, and λ is the Lagrangian multiplier controlling R-D trade-off.
Revised: In bit allocation task, bitrate is allocated to different frames/regions to minimize R-D cost R + λD , where R is total bitrate, D is total distortion, and λ is the Lagrangian multiplier controlling R-D trade-off.

Original: Li et al.
Revised: Li et al.

Original: (2022) are the pioneer of bit allocation for NVC, who improve the empirical R-D (Rate-Distortion) model from traditional video codec (Li et al., 2014; 2016) and solve the per-frame Lagrangian multiplier λ .
Revised: (2022) are the pioneer of bit allocation for NVC, who improve the empirical R-D (Rate-Distortion) model from traditional video codec (Li et al., 2014; 2016) and solve the per-frame Lagrangian multiplier λ .

Original: Other concurrent works adopt simple heuristics for coarse bit allocation (Cetin et al., 2022; Hu et al., 2022).
Revised: Other concurrent works adopt simple heuristics for coarse bit allocation (Cetin et al., 2022; Hu et al., 2022).

Original: Most recently, BAO (Bit Allocation using Optimization)
Revised: Most recently, BAO (Bit Allocation using Optimization)

Original: (Xu et al., 2022) proposes to formulate bit allocation as semi-amortized variational inference (SAVI)
Revised: (Xu et al., 2022) proposes to formulate bit allocation as semi-amortized variational inference (SAVI)

Original: (Kim et al., 2018; Marino et al., 2018) and solves it by gradient-based optimization.
Revised: (Kim et al., 2018; Marino et al., 2018) and solves it by gradient-based optimization.

Original: Specifically, it directly optimizes the variational posterior parameter to be quantized and encoded by gradient ascent, aiming at maximizing the minus overall
Revised: Specifically, it directly optimizes the variational posterior parameter to be quantized and encoded by gradient ascent, aiming at maximizing the minus overall

Original: R-D cost, which is also the evident lowerbound (ELBO).
Revised: R-D cost, which is also the evident lowerbound (ELBO).

Original: BAO does not rely on any empirical RD model and thus outperforms previous work.
Revised: BAO does not rely on any empirical RD model and thus outperforms previous work.

Original: Further, BAO shows its optimality by proving its equivalence to bit allocation with precise R-D model.
Revised: Further, BAO shows its optimality by proving its equivalence to bit allocation with precise R-D model.

Original: In this paper, we first show that BAO (Xu et al., 2022) is in fact, sub-optimal due to its implementation.
Revised: In this paper, we first show that BAO (Xu et al., 2022) is in fact, sub-optimal due to its implementation.

Original: Specifically, we find that it abuses SAVI (Kim et al., 2018;
Revised: Specifically, we find that it abuses SAVI (Kim et al., 2018;

Original: Marino et al., 2018) on latent with non-factorized variational posterior, which brings incorrect gradient signal during optimization.
Revised: Marino et al., 2018) on latent with non-factorized variational posterior, which brings incorrect gradient signal during optimization.

Original: To solve this problem, we first extend SAVI to non-factorized latent by back-propagating through gradient ascent (Domke, 2012).
Revised: To solve this problem, we first extend SAVI to non-factorized latent by back-propagating through gradient ascent (Domke, 2012).

Original: Then based on that, we correct the sub-optimal bit allocation in BAO to produce true optimal bit allocation for NVC.
Revised: Then based on that, we correct the sub-optimal bit allocation in BAO to produce true optimal bit allocation for NVC.

Original: Furthermore, we propose a computational feasible approximation to such correct but intractable bit allocation method.
Revised: Furthermore, we propose a computational feasible approximation to such correct but intractable bit allocation method.

Original: And we show that our approximation outperforms the incorrect bit allocation (BAO) in terms of R-D performance and bitrate error, and performs better than all other bit allocation methods.
Revised: And we show that our approximation outperforms the incorrect bit allocation (BAO) in terms of R-D performance and bitrate error, and performs better than all other bit allocation methods.

Original: To summarize, our contributions are as follows:
Revised: To summarize, our contributions are as follows:

Original: • We demonstrate that a previously claimed optimal bit allocation method is actually suboptimal.
Revised: • We demonstrate that a previously claimed optimal bit allocation method is actually suboptimal.

Original: We find that its sub-optimality comes from the improper application of SAVI to non-factorized latent.
Revised: We find that its sub-optimality comes from the improper application of SAVI to non-factorized latent.

Original: • We present the correct way to conduct SAVI on non-factorized latent by recursively applying back-propagation through gradient ascent.
Revised: • We present the correct way to conduct SAVI on non-factorized latent by recursively applying back-propagation through gradient ascent.

Original: Based on this, we derive the corrected optimal bit allocation algorithm for NVC.
Revised: Based on this, we derive the corrected optimal bit allocation algorithm for NVC.

Original: Furthermore, we propose a computational efficient approximation of the optimal bit allocation to make it feasible.
Revised: Furthermore, we propose a computational efficient approximation of the optimal bit allocation to make it feasible.

Original: Our proposed approach improves the R-D performance and bitrate error over the incorrect bit allocation (BAO), and outperforms all other bit allocation methods for NVC.
Revised: Our proposed approach improves the R-D performance and bitrate error over the incorrect bit allocation (BAO), and outperforms all other bit allocation methods for NVC.

Original:  P RELIMINARIES 2. N EURAL V IDEO C OMPRESSION
Revised:  P RELIMINARIES 2. N EURAL V IDEO C OMPRESSION

Original: The input of NVC is a GoP (Group of Picture) x 1: T , where x i ∈
Revised: The input of NVC is a GoP (Group of Picture) x 1: T , where x i ∈

Original: R H
Revised: R H

Original: × W is the i th frame with
Revised: × W is the i th frame with

Original: H × W pixels, and T is the number of frame inside the GoP. Most of the works in NVC follow a latent variable model with temporal autoregressive relationship (Yang et al., 2020a).
Revised: H × W pixels, and T is the number of frame inside the GoP. Most of the works in NVC follow a latent variable model with temporal autoregressive relationship (Yang et al., 2020a).

Original: Specifically, to encode x i , we first extract the motion latent w i =
Revised: Specifically, to encode x i , we first extract the motion latent w i =

Original: f wϕ ( x i , x ′ i ) from current frame x i and previous reconstructed frame x ′ i − 1 , where f wϕ ( · ) is the motion encoder parameterized by ϕ 1 .
Revised: f wϕ ( x i , x ′ i ) from current frame x i and previous reconstructed frame x ′ i − 1 , where f wϕ ( · ) is the motion encoder parameterized by ϕ 1 .

Original: Then, we encode the quantized latent ˜ w i =
Revised: Then, we encode the quantized latent ˜ w i =

Original: ⌊ w i ⌉ with the probability mass function (pmf) estimator
Revised: ⌊ w i ⌉ with the probability mass function (pmf) estimator

Original: P θ ( ˜ w i | ˜ w <i , ˜ y <i ) parameterized by θ , where ⌊·⌉ is the rounding.
Revised: P θ ( ˜ w i | ˜ w <i , ˜ y <i ) parameterized by θ , where ⌊·⌉ is the rounding.

Original: Then, we obtain the residual latent y i =
Revised: Then, we obtain the residual latent y i =

Original: f yϕ ( x , x ′ , ˜ w ) , where f yϕ ( · ) is the residual encoder.
Revised: f yϕ ( x , x ′ , ˜ w ) , where f yϕ ( · ) is the residual encoder.

Original: Then, similar to how we treat w i , we encode the quantized latent ˜ y
Revised: Then, similar to how we treat w i , we encode the quantized latent ˜ y

Original: i =
Revised: i =

Original: ⌊ y
Revised: ⌊ y

Original: i ⌉ with pmf P θ
Revised: i ⌉ with pmf P θ

Original: ( ˜ y i | ˜ w ≤ i , ˜ y <i ) .
Revised: ( ˜ y i | ˜ w ≤ i , ˜ y <i ) .

Original: Finally, we obtain the reconstructed frame x ′ i =
Revised: Finally, we obtain the reconstructed frame x ′ i =

Original: g xθ ( x ′ i − 1 , ˜ w i , ˜ y i ) , where g xθ ( · ) is the decoder parameterized by θ .
Revised: g xθ ( x ′ i − 1 , ˜ w i , ˜ y i ) , where g xθ ( · ) is the decoder parameterized by θ .

Original: As only the motion latent ˜ w i and residual latent ˜
Revised: As only the motion latent ˜ w i and residual latent ˜

Original: y
Revised: y

Original: i exist in the bitstream, the above process can be simplified as Eq.
Revised: i exist in the bitstream, the above process can be simplified as Eq.

Original: 1 and Eq.
Revised: 1 and Eq.

Original: 2, where f ϕ ( · ) is the generalized encoder and g θ ( · ) is the generalized decoder.
Revised: 2, where f ϕ ( · ) is the generalized encoder and g θ ( · ) is the generalized decoder.

Original: The target of NVC is to minimize the per-frame R-D cost R
Revised: The target of NVC is to minimize the per-frame R-D cost R

Original: i + λ i D i
Revised: i + λ i D i

Original: (Eq. 3), where R i is the bitrate, D i is the distortion and λ i is the Lagrangian multiplier controlling R-D trade-off.
Revised: (Eq. 3), where R i is the bitrate, D i is the distortion and λ i is the Lagrangian multiplier controlling R-D trade-off.

Original: The bitrate R i and distortion D i is computed as Eq. 2, where d ( · , · ) is the distortion metric.
Revised: The bitrate R i and distortion D i is computed as Eq. 2, where d ( · , · ) is the distortion metric.

Original: And λ i D i can be further interpreted as the data likelihood term − log p θ ( x i | ˜ w ≤ i , ˜ y ≤ i ) so long as we treat λ i D i as the energy function of a Gibbs distribution (Minnen et al., 2018).
Revised: And λ i D i can be further interpreted as the data likelihood term − log p θ ( x i | ˜ w ≤ i , ˜ y ≤ i ) so long as we treat λ i D i as the energy function of a Gibbs distribution (Minnen et al., 2018).

Original: Specifically, when d ( · , · ) is MSE, we can interpret λ i D i = − log p θ ( x i | ˜ w ≤ i , ˜ y ≤ i ) + const , where p θ ( x i | ˜ w ≤ i , ˜ y ≤ i ) is a Gaussian distribution N
Revised: Specifically, when d ( · , · ) is MSE, we can interpret λ i D i = − log p θ ( x i | ˜ w ≤ i , ˜ y ≤ i ) + const , where p θ ( x i | ˜ w ≤ i , ˜ y ≤ i ) is a Gaussian distribution N

Original: ( ˆ x i , 1 / 2 λ i I ) .
Revised: ( ˆ x i , 1 / 2 λ i I ) .

Original: [Equation]
Revised: [Equation]

Original: ˜
Revised: ˜

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: ˜
Revised: ˜

Original: [Equation]
Revised: [Equation]

Original: On the other hand, NVC is also closely related to Variational Autoencoder (VAE) (Kingma &
Revised: On the other hand, NVC is also closely related to Variational Autoencoder (VAE) (Kingma &

Original: Welling, 2013).
Revised: Welling, 2013).

Original: As the rounding ⌊·⌉ is not differentiable, Ball ´ e et al.
Revised: As the rounding ⌊·⌉ is not differentiable, Ball ´ e et al.

Original: ; Theis et al.
Revised: ; Theis et al.

Original: (2017) propose to relax it by additive uniform noise (AUN), and replace ˜ w i =
Revised: (2017) propose to relax it by additive uniform noise (AUN), and replace ˜ w i =

Original: ⌊ w i ⌉ , ˜ y
Revised: ⌊ w i ⌉ , ˜ y

Original: i =
Revised: i =

Original: ⌊ y
Revised: ⌊ y

Original: i ⌉ with ˜ w i =
Revised: i ⌉ with ˜ w i =

Original: w i + U ( − 0 .
Revised: w i + U ( − 0 .

Original: 5 , 0 . 5) , ˜ y
Revised: 5 , 0 . 5) , ˜ y

Original: i =
Revised: i =

Original: y i
Revised: y i

Original: + U ( − 0 .
Revised: + U ( − 0 .

Original: Under such formulation, the above encodingdecoding process becomes a VAE on graphic model ˜ w ≤ i , ˜ y ≤ i → x i with variational posterior as
Revised: Under such formulation, the above encodingdecoding process becomes a VAE on graphic model ˜ w ≤ i , ˜ y ≤ i → x i with variational posterior as

Original: Eq. 4, where w i , y i plays the role of variational posterior parameter.
Revised: Eq. 4, where w i , y i plays the role of variational posterior parameter.

Original: Then, minimizing the overall
Revised: Then, minimizing the overall

Original: R-D cost (Eq. 3) is equivalent to maximizing the evident lowerbound (ELBO)
Revised: R-D cost (Eq. 3) is equivalent to maximizing the evident lowerbound (ELBO)

Original: (Eq. 5).
Revised: (Eq. 5).

Original: q
Revised: q

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: 2. B IT A LLOCATION FOR N EURAL V IDEO C OMPRESSION
Revised: 2. B IT A LLOCATION FOR N EURAL V IDEO C OMPRESSION

Original: It is well known to video coding community that using the same R-D trade-off parameter
Revised: It is well known to video coding community that using the same R-D trade-off parameter

Original: λ
Revised: λ

Original: i to optimize R-D cost in Eq.
Revised: i to optimize R-D cost in Eq.

Original: 3 for all T frames inside a GoP is suboptimal (Li et al., 2014; 2016).
Revised: 3 for all T frames inside a GoP is suboptimal (Li et al., 2014; 2016).

Original: This sub-optimality comes from the frame reference structure and is explained in detail by Li et al.
Revised: This sub-optimality comes from the frame reference structure and is explained in detail by Li et al.

Original: ; Xu et al. (2022).
Revised: ; Xu et al. (2022).

Original: The target of bit allocation is to maximize the minus of overall R-D cost
Revised: The target of bit allocation is to maximize the minus of overall R-D cost

Original: (ELBO) L as Eq.
Revised: (ELBO) L as Eq.

Original: 6 given the overall R-D trade-off parameter λ 0 , instead of maximizing L i of each frame i separately.
Revised: 6 given the overall R-D trade-off parameter λ 0 , instead of maximizing L i of each frame i separately.

Original: The pioneer work of bit allocation in NVC (Li et al., 2022) follows bit allocation for traditional video codec (Li et al., 2016).
Revised: The pioneer work of bit allocation in NVC (Li et al., 2022) follows bit allocation for traditional video codec (Li et al., 2016).

Original: Specifically, it adopts empirical models to approximate the relationship of the rate dependency ∂R
Revised: Specifically, it adopts empirical models to approximate the relationship of the rate dependency ∂R

Original: i +1 /∂R
Revised: i +1 /∂R

Original: i and distortion dependency ∂D
Revised: i and distortion dependency ∂D

Original: i +1 /∂D
Revised: i +1 /∂D

Original: i between frames.
Revised: i between frames.

Original: Then it takes those models into Eq.
Revised: Then it takes those models into Eq.

Original: 6 to solve λ ∗ 1: T explicitly as Eq. 7.
Revised: 6 to solve λ ∗ 1: T explicitly as Eq. 7.

Original: left .
Revised: left .

Original: However, its performance heavily relies on the accuracy of empirical models.
Revised: However, its performance heavily relies on the accuracy of empirical models.

Original: [Equation]
Revised: [Equation]

Original: On the other hand, BAO (Xu et al., 2022) does not solve λ ∗ 1: T explicitly.
Revised: On the other hand, BAO (Xu et al., 2022) does not solve λ ∗ 1: T explicitly.

Original: Instead, it adopts
Revised: Instead, it adopts

Original: SAVI (Kim et al., 2018; Marino et al., 2018) to achieve implicit bit allocation.
Revised: SAVI (Kim et al., 2018; Marino et al., 2018) to achieve implicit bit allocation.

Original: To be specific, it initializes the variational posterior parameter w 

Revised: To be specific, it initializes the variational posterior parameter w

Original: : T , y 01: T from fully amortized variational inference (FAVI) as Eq.
Revised: : T , y 01: T from fully amortized variational inference (FAVI) as Eq.

Original: Then, it optimizes w 1: T , y 1: T via gradient ascent to maximize L as
Revised: Then, it optimizes w 1: T , y 1: T via gradient ascent to maximize L as

Original: Eq. 7.
Revised: Eq. 7.

Original: right .
Revised: right .

Original: During this procedure, no empirical model is required.
Revised: During this procedure, no empirical model is required.

Original: BAO further proofs that optimizing Eq.
Revised: BAO further proofs that optimizing Eq.

Original: right is equivalent to optimizing Eq.
Revised: right is equivalent to optimizing Eq.

Original: left with precise rate and distortion dependency model ∂R
Revised: left with precise rate and distortion dependency model ∂R

Original: i +1 /∂R
Revised: i +1 /∂R

Original: i , ∂D
Revised: i , ∂D

Original: i +1 /∂D
Revised: i +1 /∂D

Original: i (See Thm.
Revised: i (See Thm.

Original: 1, Thm. 2 in Xu et al.
Revised: 1, Thm. 2 in Xu et al.

Original: Thus, BAO claims that it is optimal assuming gradient ascent achieves global maximum.
Revised: Thus, BAO claims that it is optimal assuming gradient ascent achieves global maximum.

Original: However, in next section, we show that BAO (Xu et al., 2022) is in fact suboptimal due to its implementation.
Revised: However, in next section, we show that BAO (Xu et al., 2022) is in fact suboptimal due to its implementation.

Original:  W HY BAO IS S UP - OPTIMAL
Revised:  W HY BAO IS S UP - OPTIMAL

Original: BAO (Xu et al., 2022) achieves the SAVI (Kim et al., 2018; Marino et al., 2018) target in Eq.
Revised: BAO (Xu et al., 2022) achieves the SAVI (Kim et al., 2018; Marino et al., 2018) target in Eq.

Original: right by gradient-based optimization.
Revised: right by gradient-based optimization.

Original: More specifically, its update rule is described as Eq.
Revised: More specifically, its update rule is described as Eq.

Original: 8 and Eq. 9, where K is the total number of gradient ascent steps, and w ki , y ki is the posterior parameter w i ,
Revised: 8 and Eq. 9, where K is the total number of gradient ascent steps, and w ki , y ki is the posterior parameter w i ,

Original: y i after k steps of gradient ascent.
Revised: y i after k steps of gradient ascent.

Original: In the original paper of BAO, the authors also find that directly optimizing w i , y i simultaneously by Eq.
Revised: In the original paper of BAO, the authors also find that directly optimizing w i , y i simultaneously by Eq.

Original: 8 and Eq.
Revised: 8 and Eq.

Original: 9 performs worse than optimizing y i alone using Eq. 9, but they have not offered any explanation.
Revised: 9 performs worse than optimizing y i alone using Eq. 9, but they have not offered any explanation.

Original: It is obvious that optimizing y i alone is sub-optimal.
Revised: It is obvious that optimizing y i alone is sub-optimal.

Original: However, it is not obvious why jointly optimizing w i , y i with Eq.
Revised: However, it is not obvious why jointly optimizing w i , y i with Eq.

Original: 8 and Eq.
Revised: 8 and Eq.

Original: 9 fails.
Revised: 9 fails.

Original: [Equation]
Revised: [Equation]

Original: , where d L ( w k 1: T
Revised: , where d L ( w k 1: T

Original: [Equation]
Revised: [Equation]

Original: y k
Revised: y k

Original: [Equation]
Revised: [Equation]

Original: T
Revised: T

Original: [Equation]
Revised: [Equation]

Original: d
Revised: d

Original: [Equation]
Revised: [Equation]

Original: , where d L ( w k
Revised: , where d L ( w k

Original: [Equation]
Revised: [Equation]

Original: T
Revised: T

Original: [Equation]
Revised: [Equation]

Original: y
Revised: y

Original: [Equation]
Revised: [Equation]

Original: T
Revised: T

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: In fact, the update rule in Eq.
Revised: In fact, the update rule in Eq.

Original: 8 and Eq.
Revised: 8 and Eq.

Original: 9 is exactly the SAVI (Kim et al., 2018;
Revised: 9 is exactly the SAVI (Kim et al., 2018;

Original: Marino et al., 2018) when w i , y i fully factorizes (e.g. the full factorization used in mean-field (Blei et al., 2017)).
Revised: Marino et al., 2018) when w i , y i fully factorizes (e.g. the full factorization used in mean-field (Blei et al., 2017)).

Original: However, in NVC the w i , y i has complicated auto-regressive relationships (See Eq. 1 and Fig.
Revised: However, in NVC the w i , y i has complicated auto-regressive relationships (See Eq. 1 and Fig.

Original: 1.(a)).
Revised: 1.(a)).

Original: Abusing SAVI on non-factorized latent causes gradient error in two aspects: (1).
Revised: Abusing SAVI on non-factorized latent causes gradient error in two aspects: (1).

Original: The total derivative d L /d w i , d L /d y
Revised: The total derivative d L /d w i , d L /d y

Original: i is incomplete.
Revised: i is incomplete.

Original: The total derivative d L /d w i , d L /d y i and partial derivative ∂ L j /∂ w i , ∂ L j /∂ y i is evaluated at wrong value.
Revised: The total derivative d L /d w i , d L /d y i and partial derivative ∂ L j /∂ w i , ∂ L j /∂ y i is evaluated at wrong value.

Original: In next two sections, we elaborate those two issues with w i related equations in main text and y i related equations in Appendix.
Revised: In next two sections, we elaborate those two issues with w i related equations in main text and y i related equations in Appendix.

Original: A.2.
Revised: A.2.

Original: 3. I NCOMPLETE T OTAL DERIVATIVE EVALUATION
Revised: 3. I NCOMPLETE T OTAL DERIVATIVE EVALUATION

Original: According to the latent generation procedure described by Eq. 1 and Eq. 2, we draw the computational graph to describe the latent dependency as Fig. 1.(a).
Revised: According to the latent generation procedure described by Eq. 1 and Eq. 2, we draw the computational graph to describe the latent dependency as Fig. 1.(a).

Original: Based on that, we expand the total derivative d L /d w i , d L /d
Revised: Based on that, we expand the total derivative d L /d w i , d L /d

Original: y
Revised: y

Original: i as Eq.
Revised: i as Eq.

Original: 10 and Eq. 22.
Revised: 10 and Eq. 22.

Original: [Equation]

Original: ignored by BAO

Original: [Equation]
Revised: [Equation]

Original: As shown in Eq.
Revised: As shown in Eq.

Original: 8, Eq. 9 and Fig. 1.(b), BAO (Xu et al., 2022) treats the total derivative d L /d w i , d L /d
Revised: 8, Eq. 9 and Fig. 1.(b), BAO (Xu et al., 2022) treats the total derivative d L /d w i , d L /d

Original: y
Revised: y

Original: i as the sum of the frame level
Revised: i as the sum of the frame level

Original: partial derivative ∂ L j /∂ w i , ∂ L j /∂ y i , which is the direct contribution of frame i th latent w i , y i to j th frame’s R-D cost L j (as marked in Eq.
Revised: partial derivative ∂ L j /∂ w i , ∂ L j /∂ y i , which is the direct contribution of frame i th latent w i , y i to j th frame’s R-D cost L j (as marked in Eq.

Original: 10 and
Revised: 10 and

Original: Eq. 22).
Revised: Eq. 22).

Original: This incomplete evaluation of gradient signal brings sub-optimality.
Revised: This incomplete evaluation of gradient signal brings sub-optimality.

Original: Further, it is not possible to correct BAO by simply including other parts of gradient into consideration.
Revised: Further, it is not possible to correct BAO by simply including other parts of gradient into consideration.

Original: As BAO jointly updates all the latent w 1: T , y 1: T , the relationship of Eq. 2 only holds for the initial latent parameters w 01: T , y 01: T produced by FAVI.
Revised: As BAO jointly updates all the latent w 1: T , y 1: T , the relationship of Eq. 2 only holds for the initial latent parameters w 01: T , y 01: T produced by FAVI.

Original: And this important relationship is broken for parameters w k 1: T , y k 1: T after k ≥ 1 steps of update.
Revised: And this important relationship is broken for parameters w k 1: T , y k 1: T after k ≥ 1 steps of update.

Original: 3. I NCORRECT V ALUE TO E VALUATE G RADIENT
Revised: 3. I NCORRECT V ALUE TO E VALUATE G RADIENT

Original: As shown in Eq.
Revised: As shown in Eq.

Original: 8 and Eq. 9, BAO (Xu et al., 2022) simultaneously updates all the posterior parameter w 1: T , y 1: T with gradient evaluated at the same gradient ascent step w k 1: T , y k 1: T .
Revised: 8 and Eq. 9, BAO (Xu et al., 2022) simultaneously updates all the posterior parameter w 1: T , y 1: T with gradient evaluated at the same gradient ascent step w k 1: T , y k 1: T .

Original: However, as we show later in Sec. 4.1 and Fig. 1.(c), this is sub-optimal as all the descendant latent w >i , y ≥ i of w i should already complete all K steps of gradient ascent before the gradient of w i is evaluated.
Revised: However, as we show later in Sec. 4.1 and Fig. 1.(c), this is sub-optimal as all the descendant latent w >i , y ≥ i of w i should already complete all K steps of gradient ascent before the gradient of w i is evaluated.

Original: Moreover, w >i , y ≥ i should be initialized by FAVI using precedents latent.
Revised: Moreover, w >i , y ≥ i should be initialized by FAVI using precedents latent.

Original: Similar rule applies to y i .
Revised: Similar rule applies to y i .

Original: Specifically, the correct value to evaluate the gradient is as Eq.
Revised: Specifically, the correct value to evaluate the gradient is as Eq.

Original: 11 and Eq. 23, where w k i i denotes the latent w i after k
Revised: 11 and Eq. 23, where w k i i denotes the latent w i after k

Original: i steps of update, and y k ′
Revised: i steps of update, and y k ′

Original: j i denotes the latent
Revised: j i denotes the latent

Original: y i after k ′ i steps of update.
Revised: y i after k ′ i steps of update.

Original: [Equation]
Revised: [Equation]

Original: Similar to the incomplete total derivative evaluation, this problem does not have a simple solution.
Revised: Similar to the incomplete total derivative evaluation, this problem does not have a simple solution.

Original: In next section, we show how to correct both of the above-mentioned issues by recursively applying back-propagating through gradient ascent (Domke, 2012).
Revised: In next section, we show how to correct both of the above-mentioned issues by recursively applying back-propagating through gradient ascent (Domke, 2012).

Original:  C ORRECTING THE S UB - OPTIMAL B IT A LLOCATION
Revised:  C ORRECTING THE S UB - OPTIMAL B IT A LLOCATION

Original: In this section, we first extend the generic SAVI Kim et al.
Revised: In this section, we first extend the generic SAVI Kim et al.

Original: ; Marino et al.
Revised: ; Marino et al.

Original: (2018) to 2-level non-factorized latent.
Revised: (2018) to 2-level non-factorized latent.

Original: Then we further extend this result to latent with any dependency that can be described by a DAG (Directed Acyclic Graph).
Revised: Then we further extend this result to latent with any dependency that can be described by a DAG (Directed Acyclic Graph).

Original: And finally, we correct the sub-optimal bit allocation by applying the result in DAG latent to NVC.
Revised: And finally, we correct the sub-optimal bit allocation by applying the result in DAG latent to NVC.

Original: 4. SAVI ON 2- LEVEL NON - FACTORIZED LATENT
Revised: 4. SAVI ON 2- LEVEL NON - FACTORIZED LATENT

Original: In this section, we extend the SAVI on 1-level latent (Kim et al., 2018) to 2-level non-factorized latent.
Revised: In this section, we extend the SAVI on 1-level latent (Kim et al., 2018) to 2-level non-factorized latent.

Original: We denote x as evidence, w as the variational posterior parameter of the first level latent ˜ w , y as the variational posterior parameter of the second level latent ˜ y , and the ELBO to maximize as
Revised: We denote x as evidence, a as the variational posterior parameter of the first level latent ˜ a , b as the variational posterior parameter of the second level latent ˜ b , and the ELBO to maximize as

Original: L ( w , y ) .
Revised: L ( a , b ) .

Original: The posterior q ( ˜ w , ˜ y | x ) factorizes as q ( ˜ w | x ) q ( ˜ y | ˜ w , x ) , which means that y depends on w .
Revised: The posterior q ( ˜ a , ˜ b | x ) factorizes as q ( ˜ a | x ) q ( ˜ b | ˜ a , x ) , which means that b depends on a .

Original: Given w is fixed, we can directly follow Kim et al.
Revised: Given a is fixed, we can directly follow Kim et al.

Original: ; Marino et al.
Revised: ; Marino et al.

Original: (2018) to optimize y to maximize ELBO by SAVI.
Revised: (2018) to optimize b to maximize ELBO by SAVI.

Original: However, it requires some tricks to optimize w .
Revised: However, it requires some tricks to optimize a .

Original: 1 procedure solve-2-level x ,  ) initialize w 0 ← f  ) from FAVI for 
Revised: 1 procedure solve-2-level( x , a k ) initialize a 0 ← f ( x ) from FAVI for k = 0 , ..., K − 1 do d L ( a k , b K )

Original: 0 , ..., K − 1 do d L ( w k  , y K ) d w k= grad level
Revised: d a k = grad-2-level ( x , a k ) a k +1 ← a k

Original: ) w k +1 ← w k + α d L w k , y K )
Revised: [Figure] α d L ( a k , b K )

Original: d w k return w K , y K 7 procedure grad level y
Revised: d a k return a K , b K 7 procedure grad-2-level( x , a k )

Original: [Figure]
Revised: [Figure]

Original:  from FAVI fork ′ =
Revised: f ( x , a k ) from FAVI for

Original: [Figure]
Revised: [Figure]

Original: K − do 
Revised: = [Figure] 1 do [Figure]

Original: [Figure]
Revised: [Figure]

Revised: =

Original: [Figure]
Revised: [Figure]

Original: for
Revised: , 0 do

Original: [Figure] 
Revised: [Figure] [Figure]

Revised: procedure solve

Original: [Figure]
Revised: [Figure]

Original: 0 do
Revised: dag( x )

Original: [Figure]
Revised: [Figure]

Revised: in topological order

Revised: [Figure]

Revised: with

Revised: [Figure]

Revised: j

Original: [Figure]
Revised: [Figure]

Revised: ∅

Original: [Figure] 
Revised: [Figure] [Figure] [Figure] [Figure]

Original: 1 procedure solve- ) sort y y N topological order for y j
Revised: grad [Figure] dag( x

Original: P =
Revised: [Figure]

Original: [-add y j fake node y 
-]
Revised: [-add y j fake node y 
-]a k [Figure] [Figure]

Original: ’s i ) for y j ∈ C ( y i ) in topological order do y 

Revised: a k i i ) for a j ∈ C ( a i ) in topological order do a

Original: j ← f ( x , y k 0 0 , ..., y k <j <j ) from FAVI for k j = 0 , ..., K − 1 do d L ( y k 00 ,..., y kjj , y K>j ) d y kjj ← grad-dag ( x , y k 0 0 , ..., y k j j ) 
Revised: j ← f ( x , a k 0 0 , ..., a k <j <j ) from FAVI for k j = 0 , ..., K − 1 do d L ( a k 00 ,..., a kjj , a K>j ) d a kjj ← grad-dag ( x , [Equation] k 0 0 , ..., a k j j ) a k j +1 j ← a k j

Original: y k 
Revised: [Equation] [Equation] k 00 ,..., a [Equation]

Original: j +1 j ← y k j j + α d L ( y k 00 ,..., y kjj , y K>j ) d y k j j ←− y i ← ∂ L ( y k 00 ,..., y kii , y K>i ) 
Revised: , a K>j ) d a k j j ←− a i ← ∂ L ( a k 00 ,..., a kii , a K>i ) ∂ a kii for a j ∈ C ( a i ) do ←− a j ← 0 , [Equation] a

Original: ∂ y kii for y j ∈ C ( y i ) do ←− y j ← 0 , ←− y Kj ← d L ( y k 00 ,..., y kii , y K>i ) 
Revised: [Equation] ← d L ( a k 00 ,..., a kii , a K>i ) d a Kj for k j = K − 1 , ..., 0 do [Equation]

Original: d y Kj for k j = K − 1 , ..., 0 do ←− y j ← ←− y j + α ∂ 2 L ( y k 00 ,..., y kjj , y K>j ) ∂ y kii
Revised: [Equation] ..., a [Equation] [Equation]

Original:  ∂ y kjj ←−−−
Revised: j ) ∂ a [Equation] a

Revised: [Equation]

Original: y k j +1 j ←− y k j j ←←−−− y k j +1 j + α ∂ 2 L ( y k 00 ,..., y kjj , y K>j ) ∂ y kjj
Revised: [Equation] a k j [Equation] ←

Original: ∂
Revised: [Equation] j [Equation] [Equation]

Original: y kjj ←−−−
Revised: [Equation] a [Equation] , a

Original: y k j +1 j ←− y i ← ←− y i + ←− y j + ∂ [-y 
-]
Revised: [Equation] j ) ∂ [-y 
-]a [Equation]

Original: j ∂ [-y kii ←− y 
-]
Revised: ∂ [-y kii ←− y 
-][Equation] [Equation] ∂

Revised: [Equation]

Revised: [Equation]

Original: j return d L ( y k 00 ,..., y kii , y K>i )
Revised: ∂ [Equation] [Equation] return d L ( a k 00 ,..., a kii , a K>i )

Original: d y kii
Revised: d a kii

Original: =
Revised: =

Original: ←− y i
Revised: ←− a i

Original: The intuition is, we do not want to find a w that maximizes L ( w , y ) given a fixed y (or we have the gradient issue described in Sec. 3).
Revised: The intuition is, we do not want to find a a that maximizes L ( a , b ) given a fixed b (or we have the gradient issue described in Sec. 3).

Original: Instead, we want to find a w , whose max y L ( w , y ) is maximum.
Revised: Instead, we want to find a a , whose max b L ( a , b ) is maximum.

Original: This translates to the optimization problem as Eq.
Revised: This translates to the optimization problem as Eq.

Original: In fact, Eq.
Revised: In fact, Eq.

Original: 12 is a variant of setup in backpropagating through gradient ascent (Samuel & Tappen, 2009; Domke, 2012).
Revised: 12 is a variant of setup in backpropagating through gradient ascent (Samuel & Tappen, 2009; Domke, 2012).

Original: The difference is, our w also contributes directly to optimization target L ( w , y ) .
Revised: The difference is, our a also contributes directly to optimization target L ( a , b ) .

Original: From this perspective, Eq.
Revised: From this perspective, Eq.

Original: 12 is more closely connected to Kim et al.
Revised: 12 is more closely connected to Kim et al.

Original: (2018), if we treat w as the model parameter and y as latent.
Revised: (2018), if we treat a as the model parameter and b as latent.

Original: [Equation]
Revised: [Equation]

Original:  ∗
Revised: b ∗

Original: [Equation]
Revised: [Equation]

Original:  , where y ∗ ( w
Revised: )) , where b ∗ ( a )

Original: [Equation]
Revised: [Equation]

Original: [Equation] [Equation] And as SAVI on 1-level latent (Kim et al., 2018; Marino et al., 2018), we need to solve Eq.
Revised: And as SAVI on 1-level latent (Kim et al., 2018; Marino et al., 2018), we need to solve Eq.

Original: 12 using gradient ascent.
Revised: 12 using gradient ascent.

Original: Specifically, denote α as step size (learning rate), K as the total gradient ascent steps, w k as the w after k step update, y k ′ as the y after k ′ step update, and f ( . ) as FAVI procedure generating initial posterior parameters w 0 , y 0 , the optimization problem as Eq.
Revised: Specifically, denote α as step size (learning rate), K as the total gradient ascent steps, a k as the a after k step update, b k ′ as the b after k ′ step update, and f ( . ) as FAVI procedure generating initial posterior parameters a 0 , b 0 , the optimization problem as Eq.

Original: 12 translates into the update rule as Eq. 13.
Revised: 12 translates into the update rule as Eq. 13.

Original: Eq.
Revised: Eq.

Original: 13 is the guidance for designing optimization algorithm, and it also explains why the gradient of BAO (Xu et al., 2022) is evaluated at wrong value (See Sec. 3.2).
Revised: 13 is the guidance for designing optimization algorithm, and it also explains why the gradient of BAO (Xu et al., 2022) is evaluated at wrong value (See Sec. 3.2).

Original: [Equation]
Revised: [Equation]

Original: [Equation] To solve Eq. 13, we note that although d L ( w k , y k ′ ) /d
Revised: where b 0 = f ( x , a k ) (13) To solve Eq. 13, we note that although d L ( a k , b k ′ ) /d

Original: y k ′ is directly computed, d L ( w k , y K ) /d
Revised: b k ′ is directly computed, d L ( a k , b K ) /d

Original: w k is not straightforward.
Revised: a k is not straightforward.

Original: Resorting to previous works (Samuel & Tappen, 2009; Domke, 2012) in implicit differentiation and extending the results in Kim et al.
Revised: Resorting to previous works (Samuel & Tappen, 2009; Domke, 2012) in implicit differentiation and extending the results in Kim et al.

Original: (2018) from model parameters to variational posterior parameters, we implement Eq.
Revised: (2018) from model parameters to variational posterior parameters, we implement Eq.

Original: 13 as Alg.
Revised: 13 as Alg.

Original: Specifically, we first initialize wfrom FAVI.
Revised: Specifically, we first initialize afrom FAVI.

Original: Then we conduct gradient ascent on w with gradient d L ( w k , y K ) /d
Revised: Then we conduct gradient ascent on a with gradient d L ( a k , b K ) /d

Original: w k computed from the procedure grad-2-level( x , w k ).
Revised: a k computed from the procedure grad-2-level( x , a k ).

Original: And inside grad-2-level( x , w k ), y is also updated by gradient ascent, the above procedure corresponds to Eq. 13.
Revised: And inside grad-2-level( x , a k ), b is also updated by gradient ascent, the above procedure corresponds to Eq. 13.

Original: The key of Alg.
Revised: The key of Alg.

Original: 1 is the evaluation of gradient d L ( w k , y K ) /d w k .
Revised: 1 is the evaluation of gradient d L ( a k , b K ) /d a k .

Original: Formally, we have:
Revised: Formally, we have:

Original: Theorem 1.
Revised: Theorem 1.

Original: After grad-2-level( x , w k ) of Alg.
Revised: After grad-2-level( x , a k ) of Alg.

Original: 1 executes, we have the return value
Revised: 1 executes, we have the return value

Original: d L ( w k , y K ) /d w k = ←− w .
Revised: d L ( a k , b K ) /d a k = ←− a .

Original: (See proof in Appendix.
Revised: (See proof in Appendix.

Original: A.1.)
Revised: A.1.)

Original: 4. SAVI ON DAG- DEFINED N ON - FACTORIZED L ATENT
Revised: 4. SAVI ON DAG- DEFINED N ON - FACTORIZED L ATENT

Original: In this section, we extend the result from previous section to SAVI on general non-factorized latent with dependency described by any DAG.
Revised: In this section, we extend the result from previous section to SAVI on general non-factorized latent with dependency described by any DAG.

Original: This DAG is the computational graph during network inference, and it is also the directed graphical model (DGM) (Koller & Friedman, 2009) defining the factorization of latent variables during inference.
Revised: This DAG is the computational graph during network inference, and it is also the directed graphical model (DGM) (Koller & Friedman, 2009) defining the factorization of latent variables during inference.

Original: This is the general case covering all dependency that can be described by DGM.
Revised: This is the general case covering all dependency that can be described by DGM.

Original: This extension is necessary to perform SAVI on latent with complicated dependency (e.g. bit allocation of NVC).
Revised: This extension is necessary to perform SAVI on latent with complicated dependency (e.g. bit allocation of NVC).

Original: Similar to the 2-level latent setup, we consider performing SAVI on N variational posterior parameter y 1 , ..., y N with their dependency defined by a computational graph G , i.e., their corresponding latent variable ˜ y 1 , ..., ˜ y N ’s posterior distribution factorizes as G .
Revised: Similar to the 2-level latent setup, we consider performing SAVI on N variational posterior parameter a 1 , ..., a N with their dependency defined by a computational graph G , i.e., their corresponding latent variable ˜ a 1 , ..., ˜ a N ’s posterior distribution factorizes as G .

Original: Specifically, we denote y j ∈ C ( y i ) , y i ∈ P ( y j ) if an edge exists from y i to y j .
Revised: Specifically, we denote a j ∈ C ( a i ) , a i ∈ P ( a j ) if an edge exists from a i to a j .

Original: This indicates that ˜ y j conditions on ˜ y i .
Revised: This indicates that ˜ a j conditions on ˜ a i .

Original: Without loss of generality, we assume y 1 , ..., y N is sorted in topological order.
Revised: Without loss of generality, we assume a 1 , ..., a N is sorted in topological order.

Original: This means that if y j ∈ C ( y i ) , y i ∈ P ( y j ) , then i < j .
Revised: This means that if a j ∈ C ( a i ) , a i ∈ P ( a j ) , then i < j .

Original: Each latent is optimized by K -step gradient ascent, and y k i i denotes the latent y i after k i steps of update.
Revised: Each latent is optimized by K -step gradient ascent, and a k i i denotes the latent a i after k i steps of update.

Original: Then, similar to 2-level latent, we have the update rule as Eq. 14:
Revised: Then, similar to 2-level latent, we have the update rule as Eq. 14:

Original: [Equation]
Revised: [Equation]

Original: , where y 0 >i
Revised: , where a 0 >i

Original: = f( x , y
Revised: [Equation] f

Original: [Equation]
Revised: [Equation]

Original:  1 1 , ..., y k i i
Revised: x , a k 1 1 , ..., a k i i

Original: [Equation]
Revised: [Equation]

Original: 14) , which can be translated into Alg. 2.
Revised: 14) , which can be translated into Alg. 2.

Original: Specifically, we first sort the latent in topological order.
Revised: Specifically, we first sort the latent in topological order.

Original: Then, we add a fake latent y 0 to the front of all y s. Its children are all the y s with 0 in-degree.
Revised: Then, we add a fake latent a 0 to the front of all a s. Its children are all the a s with 0 in-degree.

Original: Then, we can solve the SAVI on y 1 , ..., y N using gradient ascent by executing the procedure graddag( x , y k 0 0 , ..., y k i i ) in Alg. 2 recursively.
Revised: Then, we can solve the SAVI on a 1 , ..., a N using gradient ascent by executing the procedure graddag( x , a k 0 0 , ..., a k i i ) in Alg. 2 recursively.

Original: Inside procedure grad-dag( x , y k 0 0 , ..., y k i i ), the gradient to update y i relies on the convergence of its children y j ∈ C ( y i ) , which is implemented by the recursive depth-first search (DFS) in line 11.
Revised: Inside procedure grad-dag( x , a k 0 0 , ..., a k i i ), the gradient to update a i relies on the convergence of its children a j ∈ C ( a i ) , which is implemented by the recursive depth-first search (DFS) in line 11.

Original: And upon the completion of procedure grad-dag( x , y 00 ), all the latent converges to y K 1 , ..., y KN .
Revised: And upon the completion of procedure grad-dag( x , a 00 ), all the latent converges to a K 1 , ..., a KN .

Original: Similar to the 2-level latent case, the key of Alg.
Revised: Similar to the 2-level latent case, the key of Alg.

Original: 2 is the evaluation of gradient d L ( y k 0 0 , ..., y k
Revised: 2 is the evaluation of gradient d L ( a k 0 0 , ..., a k

Original: i i , y K>i ) /d y k
Revised: i i , a K>i ) /d a k

Original: i i .
Revised: i i .

Original: Formally, we have:
Revised: Formally, we have:

Original: Theorem 2.
Revised: Theorem 2.

Original: After the procedure grad-dag( x , y k 0 0 , ..., y k i i ) in Alg.
Revised: After the procedure grad-dag( x , a k 0 0 , ..., a k i i ) in Alg.

Original: 2 executes, we have the return value
Revised: 2 executes, we have the return value

Original: d L ( y k 0 0 , ..., y k i i , y K>i ) /d y k i i =
Revised: d L ( a k 0 0 , ..., a k i i , a K>i ) /d a k i i =

Original: ←− y i .
Revised: ←− a i .

Original: (See proof in Appendix.
Revised: (See proof in Appendix.

Original: A.1.)
Revised: A.1.)

Original: To better understand how Alg.
Revised: To better understand how Alg.

Original: 2 works, we provide a detailed example in Fig.
Revised: 2 works, we provide a detailed example in Fig.

Original: 5 of Appendix.
Revised: 5 of Appendix.

Original: A.3.
Revised: A.3.

Original: 4. C ORRECTING THE S UB - OPTIMAL B IT A LLOCATION USING SAVI ON DAG
Revised: 4. C ORRECTING THE S UB - OPTIMAL B IT A LLOCATION USING SAVI ON DAG

Original: With the result in previous section, correcting BAO (Xu et al., 2022) seems to be trivial.
Revised: With the result in previous section, correcting BAO (Xu et al., 2022) seems to be trivial.

Original: We only need to sort the latent in topological order as w 1 , y 1 , ..., w T , y T ,  and run Alg. 2 to obtain the optimized latent parameters w K 1 , y K 1 , ..., w KT , y KT .
Revised: We only need to sort the latent in topological order as w 1 , y 1 , ..., w T , y T , treat them as a 1 , ..., a 2 T + and run Alg. 2 to obtain the optimized latent parameters w K 1 , y K 1 , ..., w KT , y KT .

Original: And the gradient d L ( y k 0 0 , ..., y k i i , y K>i ) /d y k
Revised: And the gradient d L ( a k 0 0 , ..., a k i i , a K>i ) /d a k

Original: i i computed in Alg.
Revised: i i computed in Alg.

Original: 2 resolves the issue of BAO described in Sec. 
Revised: 2 resolves the issue of BAO described in Sec. 3.1 and

Original: 3.1 and Sec. 3.2.
Revised: Sec. 3.2.

Original: However, an evident problem is the temporal complexity.
Revised: However, an evident problem is the temporal complexity.

Original: Given the latent number N and gradient ascent step number K , Alg. 2 has temporal complexity of Θ( K N ) .
Revised: Given the latent number N and gradient ascent step number K , Alg. 2 has temporal complexity of Θ( K N ) .

Original: NVC with GoP size 10 has approximately N =latent, and the SAVI on NVC (Xu et al., 2022) takes around K = 2000 step to converge.
Revised: NVC with GoP sizehas approximately N = 20 latent, and the SAVI on NVC (Xu et al., 2022) takes around K =step to converge.

Original: For bit allocation, the complexity of Alg.
Revised: For bit allocation, the complexity of Alg.

Original: 2 is ≈ 2000 20 , which is intractable.
Revised: 2 is ≈ 2000 20 , which is intractable.

Original: On the other hand, BAO’s complexity is reasonable ( Θ( KN )
Revised: On the other hand, BAO’s complexity is reasonable ( Θ( KN )

Original: ≈ 4 × 10 4 ).
Revised: ≈ 4 × 10 4 ).

Original: Thus, in next section, we provide a feasible approximation to such intractable corrected bit allocation.
Revised: Thus, in next section, we provide a feasible approximation to such intractable corrected bit allocation.

Original: 4. F EASIBLE A PPROXIMATION TO THE C ORRECTED B IT A LLOCATION
Revised: 4. F EASIBLE A PPROXIMATION TO THE C ORRECTED B IT A LLOCATION

Original: In order to solve problem with practical size such as bit allocation on NVC, we provide an approximation to the SAVI (Kim et al., 2018; Marino et al., 2018) on DAG described in Sec. 4.2.
Revised: In order to solve problem with practical size such as bit allocation on NVC, we provide an approximation to the SAVI (Kim et al., 2018; Marino et al., 2018) on DAG described in Sec. 4.2.

Original: The general idea is that, when being applied to bit allocation of NVC, the accurate SAVI on DAG (Alg. 2) satisfies both requirement on gradient signal described in Sec. 3.1 and Sec. 3.2.
Revised: The general idea is that, when being applied to bit allocation of NVC, the accurate SAVI on DAG (Alg. 2) satisfies both requirement on gradient signal described in Sec. 3.1 and Sec. 3.2.

Original: We can not make it tractable without breaking them.
Revised: We can not make it tractable without breaking them.

Original: Thus, we break one of them and achieve a reasonable complexity, while maintain a superior performance compared with BAO (Xu et al., 2022).
Revised: Thus, we break one of them and achieve a reasonable complexity, while maintain a superior performance compared with BAO (Xu et al., 2022).

Original: We consider the approximation in Eq.
Revised: We consider the approximation in Eq.

Original: 15 which breaks the requirement for gradient evaluation in
Revised: 15 which breaks the requirement for gradient evaluation in

Original: Sec. 3.2.
Revised: Sec. 3.2.

Original: Based on Eq.
Revised: Based on Eq.

Original: 15 and the requirement in Sec.
Revised: 15 and the requirement in Sec.

Original: 3.1, we design an approximation of accurate
Revised: 3.1, we design an approximation of accurate

Original: SAVI as Alg. 4.
Revised: SAVI as Alg. 4.

Original: When being applied to bit allocation in NVC, it satisfies the gradient requirement in Sec.
Revised: When being applied to bit allocation in NVC, it satisfies the gradient requirement in Sec.

Original: 3.1 while maintaining a temporal complexity of Θ( KN ) as BAO.
Revised: 3.1 while maintaining a temporal complexity of Θ( KN ) as BAO.

Original: [Equation]
Revised: [Equation]

Original: Specifically, with the approximation in Eq. 15, the recurrent gradient computation in Alg. 2 becomes unnecessary as the right hand side of Eq.
Revised: Specifically, with the approximation in Eq. 15, the recurrent gradient computation in Alg. 2 becomes unnecessary as the right hand side of Eq.

Original: 15 does not require y K>i .
Revised: 15 does not require a K>i .

Original: However, to maintain the dependency of latent described in Sec. 3.1, as Alg. 2, we still need to ensure that the children node y j ∈ C ( y i ) are re-initialized by FAVI every-time when y i is updated.
Revised: However, to maintain the dependency of latent described in Sec. 3.1, as Alg. 2, we still need to ensure that the children node a j ∈ C ( a i ) are re-initialized by FAVI every-time when a i is updated.

Original: Therefore, a reasonable approach is to traverse the graph in topological order.
Revised: Therefore, a reasonable approach is to traverse the graph in topological order.

Original: We keep the children node y j untouched until all its parent node y i ∈ P ( y j ) ’s gradient ascent is completed and y Ki is known.
Revised: We keep the children node a j untouched until all its parent node a i ∈ P ( a j ) ’s gradient ascent is completed and a Ki is known.

Original: And the resulting approximate SAVI algorithm is as Alg. 4.
Revised: And the resulting approximate SAVI algorithm is as Alg. 4.

Original: When applied to bit allocation, it satisfies the gradient requirement in Sec. 3.1, and as BAO, its temporal complexity is Θ( KN ) .
Revised: When applied to bit allocation, it satisfies the gradient requirement in Sec. 3.1, and as BAO, its temporal complexity is Θ( KN ) .

Original: 1 procedure solve-bao( x ) y 01 , ..., y 0 N ← f ( x ) from FAVI for k = 0 , ..., K − 1 do for i = 1 , ..., N do y k [-+
-] i 
Revised: 1 procedure solve-bao( x ) a 01 , ..., a 0 N ← f ( x ) from FAVI for k = 0 , ..., K − 1 do for i = 1 , ..., N do a k [-+
-]+1 i ←

Original: [Equation]
Revised: [Equation]

Original: ∂ y ki return y K 1 , ..., y KN
Revised: ∂ a ki return a K 1 , ..., a KN

Original: Algorithm
Revised: Algorithm

Original: 1 procedure solve-approx-dag( x ) sort y 1 , ..., y N in topological order for i = 1 , ..., N do [-y 
-] i , ..., y 0 N ← f ( x , y K<i ) from FAVI for k = 0 , ..., K − 1 do d L ( y K<
Revised: 1 procedure solve-approx-dag( x ) sort a 1 , ..., a N in topological order for i = 1 , ..., N do [-y 
-]a 0 i , ..., a 0 N ← f ( x , a K<i ) from FAVI for k = 0 , ..., K − 1 do d L ( a K<

Original: i , y ki , y K>i )
Revised: i , a ki , a K>i )

Original: d y ki ≈ d L ( y K< i , y ki , y 0 >i )
Revised: d a ki ≈ d L ( a K<i , a ki , a 0 >i )

Original: d y ki y k [-+
-] i ← y ki + α d L ( y K<
Revised: d a ki a k [-+
-]+1 i ← a ki + α d L ( a K<

Original: i , y ki , y K>i )
Revised: i , a ki , a K>i )

Original: d y ki return y K 1 , ..., y KN
Revised: d a ki return a K 1 , ..., a KN

Original: To better understand BAO (Xu et al., 2022) in SAVI context, we rewrite it by general SAVI notation instead of NVC notation in Alg.
Revised: To better understand BAO (Xu et al., 2022) in SAVI context, we rewrite it by general SAVI notation instead of NVC notation in Alg.

Original: We highlight the difference between BAO (Alg. 3)
Revised: We highlight the difference between BAO (Alg. 3)

Original: (Xu et al., 2022), the accurate SAVI on DAG latent (Alg. 2) and the approximate SAVI on DAG latent (Alg. 4) from several aspects:
Revised: (Xu et al., 2022), the accurate SAVI on DAG latent (Alg. 2) and the approximate SAVI on DAG latent (Alg. 4) from several aspects:

Original: • Graph Traversal Order : BAO performs gradient ascent on y 1: T all together.
Revised: • Graph Traversal Order : BAO performs gradient ascent on a 1: T all together.

Original: The accurate SAVI only updates y i
Revised: The accurate SAVI only updates a i

Original: when y >
Revised: when a >

Original: i ’s update is complete and y K> i is known.
Revised: i ’s update is complete and a K>i is known.

Original: The approximate SAVI only updates y i when y
Revised: The approximate SAVI only updates a i when a <i ’s update is complete and a K

Original: <
Revised: <

Original: i ’s update is complete and y K<

Original: i is known.
Revised: i is known.

Original: • Gradient Correctness : When being applied to bit allocation in NVC, BAO violates the gradient rule in Sec.
Revised: • Gradient Correctness : When being applied to bit allocation in NVC, BAO violates the gradient rule in Sec.

Original: 3.1 and Sec. 3.2, accurate SAVI satisfies both rules, approximate SAVI satisfies Sec.
Revised: 3.1 and Sec. 3.2, accurate SAVI satisfies both rules, approximate SAVI satisfies Sec.

Original: 3.1 and violates Sec. 3.2.
Revised: 3.1 and violates Sec. 3.2.

Original: • Temporal Complexity : With the latent number N and steps of gradient ascent K , the complexity of BAO is Θ( KN ) , the complexity of accurate SAVI is Θ( K N ) and the complexity of approximate SAVI is Θ( KN ) .
Revised: • Temporal Complexity : With the latent number N and steps of gradient ascent K , the complexity of BAO is Θ( KN ) , the complexity of accurate SAVI is Θ( K N ) and the complexity of approximate SAVI is Θ( KN ) .

Original: Then we can simply apply Alg.
Revised: Then we can simply apply Alg.

Original: 4 to bit allocation in NVC to obtain a feasible approximation of the corrected optimal bit allocation.
Revised: 4 to bit allocation in NVC to obtain a feasible approximation of the corrected optimal bit allocation.

Original: And in Sec. 6.2, we empirically show that our approximation improves the R-D performance over BAO (Xu et al., 2022) with even smaller number of updates.
Revised: And in Sec. 6.2, we empirically show that our approximation improves the R-D performance over BAO (Xu et al., 2022) with even smaller number of updates.

Original:  R ELATED W ORK : B IT A LLOCATION & SAVI FOR N EURAL C OMPRESSION
Revised:  R ELATED W ORK : B IT A LLOCATION & SAVI FOR N EURAL C OMPRESSION

Original: Li et al.
Revised: Li et al.

Original: (2022) are the pioneer of bit allocation for NVC and their work is elaborated in Sec. 2.2.
Revised: (2022) are the pioneer of bit allocation for NVC and their work is elaborated in Sec. 2.2.

Original: Other recent works that consider bit allocation for NVC only adopt simple heuristic such as inserting 1 high quality frame per 4 frames (Hu et al., 2022; Cetin et al., 2022).
Revised: Other recent works that consider bit allocation for NVC only adopt simple heuristic such as inserting 1 high quality frame per 4 frames (Hu et al., 2022; Cetin et al., 2022).

Original: On the other hand, OEU (Lu et al., 2020) is also recognised as frame-level bit allocation while its performance is inferior than
Revised: On the other hand, OEU (Lu et al., 2020) is also recognised as frame-level bit allocation while its performance is inferior than

Original: BAO (Xu et al., 2022).
Revised: BAO (Xu et al., 2022).

Original: BAO is the most recent work with best R-D performance.
Revised: BAO is the most recent work with best R-D performance.

Original: It is elaborated in
Revised: It is elaborated in

Original: Sec. 2.2 and Sec. 3, and corrected in the previous section.
Revised: Sec. 2.2 and Sec. 3, and corrected in the previous section.

Original: Semi-Amortized Variational Inference (SAVI) is proposed by Kim et al.
Revised: Semi-Amortized Variational Inference (SAVI) is proposed by Kim et al.

Original: ; Marino et al.
Revised: ; Marino et al.

Original: The idea is that works following Kingma & Welling (2013) use fully amortized inference parameter ϕ for all data, which leads to the amortization gap (Cremer et al., 2018).
Revised: The idea is that works following Kingma & Welling (2013) use fully amortized inference parameter ϕ for all data, which leads to the amortization gap (Cremer et al., 2018).

Original: SAVI reduces this gap by optimizing the variational posterior parameter after initializing it with inference network.
Revised: SAVI reduces this gap by optimizing the variational posterior parameter after initializing it with inference network.

Original: It adopts back-propagating through gradient ascent (Domke, 2012) to evaluate the gradient of model parameters.
Revised: It adopts back-propagating through gradient ascent (Domke, 2012) to evaluate the gradient of model parameters.

Original: We adopt a similar method to extend SAVI to non-factorized latent.
Revised: We adopt a similar method to extend SAVI to non-factorized latent.

Original: When applying SAVI to practical neural codec, researchers abandon the nested model parameter update for efficiency.
Revised: When applying SAVI to practical neural codec, researchers abandon the nested model parameter update for efficiency.

Original: Prior works (Djelouah & Schroers, 2019; Yang et al., 2020b; Zhao et al., 2021; Gao et al., 2022) adopt
Revised: Prior works (Djelouah & Schroers, 2019; Yang et al., 2020b; Zhao et al., 2021; Gao et al., 2022) adopt

Original: SAVI to boost R-D performance and achieve variable bitrate in image compression.
Revised: SAVI to boost R-D performance and achieve variable bitrate in image compression.

Original: And BAO (Xu et al., 2022) is the first to consider SAVI for bit allocation.
Revised: And BAO (Xu et al., 2022) is the first to consider SAVI for bit allocation.

Original:  E XPERIMENTS 6. E XPERIMENTAL S ETTINGS
Revised:  E XPERIMENTS 6. E XPERIMENTAL S ETTINGS

Original: We implement our approach in PyTorch 1.9 with CUDA 11.2, and run the experiments on
Revised: We implement our approach in PyTorch 1.9 with CUDA 11.2, and run the experiments on

Original: NVIDIA(R) A100 GPU.
Revised: NVIDIA(R) A100 GPU.

Original: Most of the other settings are intentionally kept the same as BAO (Xu et al., 2022).
Revised: Most of the other settings are intentionally kept the same as BAO (Xu et al., 2022).

Original: Specifically, we adopt HEVC Common Testing Condition (CTC) (Bossen et al., 2013) and UVG dataset (Mercat et al., 2020).
Revised: Specifically, we adopt HEVC Common Testing Condition (CTC) (Bossen et al., 2013) and UVG dataset (Mercat et al., 2020).

Original: And we measure the R-D performance in BjontegaardBitrate (BD-BR) and BD-PSNR (Bjontegaard, 2001).
Revised: And we measure the R-D performance in BjontegaardBitrate (BD-BR) and BD-PSNR (Bjontegaard, 2001).

Original: For baseline NVC (Lu et al., 2019; Li et al., 2021), we adopt the official pre-trained models.
Revised: For baseline NVC (Lu et al., 2019; Li et al., 2021), we adopt the official pre-trained models.

Original: And we select target λ 

Revised: And we select target λ

Original: = { 256 , 512 , 1024 , 2048 } .
Revised: = { 256 , 512 , 1024 , 2048 } .

Original: For gradient ascent, we adopt Adam (Kingma & Ba, 2014) optimizer with lr = 1 × 10 − 3 .
Revised: For gradient ascent, we adopt Adam (Kingma & Ba, 2014) optimizer with lr = 1 × 10 − 3 .

Original: We set the gradient ascent step K = 2000 for the first frame and K = 400 for other frames.
Revised: We set the gradient ascent step K = 2000 for the first frame and K = 400 for other frames.

Original: More details are presented in Appendix.
Revised: More details are presented in Appendix.

Original: A.4.
Revised: A.5.

Original: 6. Q UANTITATIVE R ESULTS
Revised: 6. Q UANTITATIVE R ESULTS

Original: As shown in Tab.
Revised: As shown in Tab.

Original: [Table]
Revised: [Table]

Original: our method consistently improves the R-D performance in terms of BD-BR over
Revised: our method consistently improves the R-D performance in terms of BD-BR over

Original: BAO (Xu et al., 2022) on both baseline methods and all datasets.
Revised: BAO (Xu et al., 2022) on both baseline methods and all datasets.

Original: Moreover, this improvement is especially significant (more than 10% in BD-BR) when the baseline is DCVC (Li et al., 2021).
Revised: Moreover, this improvement is especially significant (more than 10% in BD-BR) when the baseline is DCVC (Li et al., 2021).

Original: And both BAO and our proposed correction significantly outperform other approaches.
Revised: And both BAO and our proposed correction significantly outperform other approaches.

Original: It is also noteworthy that with our bit allocation, DVC (the SOTA method in 2019) already outperforms DCVC (the SOTA method in 2021) by large margin (See the red solid line and black dash line in Fig. 2).
Revised: It is also noteworthy that with our bit allocation, DVC (the SOTA method in 2019) already outperforms DCVC (the SOTA method in 2021) by large margin (See the red solid line and black dash line in Fig. 2).

Original: [Table]
Revised: [Table]

Original: our approach compared with others.
Revised: our approach compared with others.

Original: 1 comes from Li et al.
Revised: 1 comes from Li et al.

Original: 2 comes from Xu et al.
Revised: 2 comes from Xu et al.

Original: Other than R-D performance, the bitrate error of our approach is also significantly smaller than BAO (Xu et al., 2022)
Revised: Other than R-D performance, the bitrate error of our approach is also significantly smaller than BAO (Xu et al., 2022)

Original: (See Tab. 2).
Revised: (See Tab. 2).

Original: The bitrate error is measured as the relative bitrate difference before and after bit allocation.
Revised: The bitrate error is measured as the relative bitrate difference before and after bit allocation.

Original: The smaller it is, the easier it is to achieve the desired bitrate accurately.
Revised: The smaller it is, the easier it is to achieve the desired bitrate accurately.

Original: For complexity, our approach only performs 920 steps of gradient ascent per-frame, while BAO requires 2000 steps.
Revised: For complexity, our approach only performs 920 steps of gradient ascent per-frame, while BAO requires 2000 steps.

Original: See more quantitative results (BD-PSNR & R-D curves) in Appendix.
Revised: See more quantitative results (BD-PSNR & R-D curves) in Appendix.

Original: A.5.
Revised: A.6.

Original: 6. A BLATION S TUDY , A NALYSIS & Q UALITATIVE R ESULTS
Revised: 6. A BLATION S TUDY , A NALYSIS & Q UALITATIVE R ESULTS

Original: Tab. 3 shows that for BAO (Xu et al., 2022), jointly optimizing w 1: T , y 1: T performs worse than optimizing y 1: T or w 1: T alone.
Revised: Tab. 3 shows that for BAO (Xu et al., 2022), jointly optimizing w 1: T , y 1: T performs worse than optimizing y 1: T or w 1: T alone.

Original: This counter-intuitive phenomena comes from its incorrect estimation of gradient signal.
Revised: This counter-intuitive phenomena comes from its incorrect estimation of gradient signal.

Original: For the proposed approach that corrects this, jointly optimizing w 1: T , y 1: T performs better than optimizing y 1: T or w 1: T alone, which is aligned with our intuition
Revised: For the proposed approach that corrects this, jointly optimizing w 1: T , y 1: T performs better than optimizing y 1: T or w 1: T alone, which is aligned with our intuition

Original: [Table]
Revised: [Table]

Original: [Table]
Revised: [Table]

Original: To better understand why our method works
Revised: To better understand why our method works

Original: [Table]
Revised: [Table]

Original: we present the R-D cost, distortion and rate versus frame/latent index for different methods in Fig.
Revised: we present the R-D cost, distortion and rate versus frame/latent index for different methods in Fig.

Original: 3: top-left shows that the R-D cost of our approach consistently decreases according to SAVI stage.
Revised: 3: top-left shows that the R-D cost of our approach consistently decreases according to SAVI stage.

Original: Moreover, it outperforms BAO after 4 th frame; top-right shows that for each frame the R-D cost of our method is lower than BAO; bottom-left shows that the distortion part of R-D cost of our approach is approximately the same as BAO.
Revised: Moreover, it outperforms BAO after 4 th frame; top-right shows that for each frame the R-D cost of our method is lower than BAO; bottom-left shows that the distortion part of R-D cost of our approach is approximately the same as BAO.

Original: While bottom-right shows that the advantage of our approach over BAO lies in the bitrate.
Revised: While bottom-right shows that the advantage of our approach over BAO lies in the bitrate.

Original: More specifically, BAO increases the bitrate of y i s after SAVI, while our correction decreases it.
Revised: More specifically, BAO increases the bitrate of y i s after SAVI, while our correction decreases it.

Original: See more analysis in Appendix.
Revised: See more analysis in Appendix.

Original: A.6 and qualitative results in Appendix.
Revised: A.9 and qualitative results in Appendix.

Original: A.7.
Revised: A.10.

Original:  D ISCUSSION & C ONCLUSION
Revised:  D ISCUSSION & C ONCLUSION

Original: Despite our correction is already more efficient than original BAO (Xu et al., 2022), its encoding speed remains far from real-time.
Revised: Despite our correction is already more efficient than original BAO (Xu et al., 2022), its encoding speed remains far from real-time.

Original: Thus, it is limited to scenarios where R-D performance matters much more than encoding time (e.g. video on demand).
Revised: Thus, it is limited to scenarios where R-D performance matters much more than encoding time (e.g. video on demand).

Original: See more discussion in Appendix.
Revised: See more discussion in Appendix.

Original: A.8.
Revised: A.11.

Original: To conclude, we show that a previous bit allocation method for NVC is sub-optimal as it abuses SAVI on non-factorized latent.
Revised: To conclude, we show that a previous bit allocation method for NVC is sub-optimal as it abuses SAVI on non-factorized latent.

Original: Then, we propose the correct SAVI on general non-factorized latent by back-propagating through gradient ascent, and we further propose a feasible approximation to make it practical for bit allocation.
Revised: Then, we propose the correct SAVI on general non-factorized latent by back-propagating through gradient ascent, and we further propose a feasible approximation to make it tractable for bit allocation.

Original: Experimental results show that our correction significantly improves the R-D performance.
Revised: Experimental results show that our correction significantly improves the R-D performance.

Original: E THICS S TATEMENT
Revised: E THICS S TATEMENT

Original: Improving the R-D performance of NVC has positive social value, in terms of reducing carbon emission by saving the resources required to transfer and store videos.
Revised: Improving the R-D performance of NVC has positive social value, in terms of reducing carbon emission by saving the resources required to transfer and store videos.

Original: Moreover, unlike traditional codecs such as H.266 (Bross et al., 2021), neural video codec does not require dedicated hardware.
Revised: Moreover, unlike traditional codecs such as H.266 (Bross et al., 2021), neural video codec does not require dedicated hardware.

Original: Instead, it can be deployed with general neural accelerators.
Revised: Instead, it can be deployed with general neural accelerators.

Original: Improving the R-D performance of
Revised: Improving the R-D performance of

Original: NVC prompts the practical deployment of video codecs that are independent of dedicated hardware, and lowers the hardware-barrier of playing multi-media contents.
Revised: NVC prompts the practical deployment of video codecs that are independent of dedicated hardware, and lowers the hardware-barrier of playing multi-media contents.

Original: R EPRODUCIBILITY S TATEMENT
Revised: R EPRODUCIBILITY S TATEMENT

Original: For theoretical results, both of the two theorems are followed by proof in Appendix.
Revised: For theoretical results, both of the two theorems are followed by proof in Appendix.

Original: A.1.
Revised: A.1.

Original: For a relatively complicated novel algorithm (Alg. 2), we provide an illustration of the step by step execution procedure in Appendix.
Revised: For a relatively complicated novel algorithm (Alg. 2), we provide an illustration of the step by step execution procedure in Appendix.

Original: A.3.
Revised: A.3.

Original: For experiment, both of the two datasets are publicly accessible.
Revised: For experiment, both of the two datasets are publicly accessible.

Original: In Appendix.
Revised: In Appendix.

Original: A.4, we provide more implementation details including all the hyper-parameters.
Revised: A.5, we provide more implementation details including all the hyper-parameters.

Original: Moreover, we provide our source code for reproducing the empirical results in supplementary material
Revised: Moreover, we provide our source code for reproducing the empirical results in supplementary material

Original: A A PPENDIX A. P ROOF OF T HM 1 AND T HM
Revised: A A PPENDIX A. P ROOF OF T HM 1 AND T HM

Original: Theorem 1.
Revised: Theorem 1.

Original: After the procedure grad-2-level( x , w k ) of Alg.
Revised: After the procedure grad-2-level( x , a k ) of Alg.

Original: 1 executes, we have the return value
Revised: 1 executes, we have the return value

Original: d L ( w k , y K ) /d w k = ←− w .
Revised: d L ( a k , b K ) /d a k = ←− a .

Original: Proof.
Revised: Proof.

Original: This proof extends the proof of Thm. 1 in Domke (2012), and it also serves as a formal justification of Alg. 1 in Kim et al.
Revised: This proof extends the proof of Thm. 1 in Domke (2012), and it also serves as a formal justification of Alg. 1 in Kim et al.

Original: Note that our paper and Kim et al.
Revised: Note that our paper and Kim et al.

Original: (2018) are subtly different from Samuel & Tappen (2009);
Revised: (2018) are subtly different from Samuel & Tappen (2009);

Original: Domke (2012) as our high level parameter w not only generate low level parameter y , but also directly contributes to optimization target (See Fig. 4).
Revised: Domke (2012) as our high level parameter w not only generate low level parameter y , but also directly contributes to optimization target (See Fig. 4).

Original: As the computational graph in Fig.
Revised: As the computational graph in Fig.

Original: 4 shows, we can expand d L ( w k , y K ) /d
Revised: 4 shows, we can expand d L ( a k , b K ) /d

Original: w k as Eq.
Revised: a k as Eq.

Original: 16, with each term solved in Eq.
Revised: 16, with each term solved in Eq.

Original: 18 and Eq. 19.
Revised: 18 and Eq. 19.

Original: [Equation]
Revised: [Equation]

Original: To solve Eq.
Revised: To solve Eq.

Original: 16, we first note that ∂ L ( w k , y K ) /∂
Revised: 16, we first note that ∂ L ( a k , b K ) /∂

Original: w k , d L ( w k , y K ) /d
Revised: a k , d L ( a k , b K ) /d

Original: y K , ∂ y 0 /∂ w k is naturally known.
Revised: b K , ∂ b 0 /∂ a k is naturally known.

Original: Then, by taking partial derivative of the update rule of gradient ascent y k ′ +1 ← y k ′ + αd L ( w k , y k ′ ) /d
Revised: Then, by taking partial derivative of the update rule of gradient ascent b k ′ +1 ← b k ′ + αd L ( a k , b k ′ ) /d

Original: y k ′ with regard to w k , y k ′ , we have Eq.
Revised: b k ′ with regard to a k , b k ′ , we have Eq.

Original: 17 and Eq.
Revised: 17 and Eq.

Original: Note that Eq.
Revised: Note that Eq.

Original: 18 is the partial derivative ∂ y k 
Revised: 18 is the partial derivative ∂ b k ′

Original: +1 /∂
Revised: +1 /∂

Original: w k instead of total derivative d y k ′
Revised: a k instead of total derivative d b k ′

Original: +1 /d 
Revised: +1 /d a k

Original: w k = ( ∂ y k 
Revised: = ( ∂ b k ′

Original: +1 /∂
Revised: +1 /∂

Original: y k ′ )
Revised: b k ′ )

Original: ( d y k ′ /d w k )
Revised: [Equation] d b [Equation]

Original: + ∂
Revised: [Equation] [Equation]

Original: y k ′
Revised: ∂ b k ′

Original: +1 /
Revised: +1 /∂

Original: [Equation]

Original:  k .
Revised: a k .

Original: [Equation]
Revised: [Equation]

Original: And those second order terms can either be directly evaluated or approximated via finite difference as Eq.
Revised: And those second order terms can either be directly evaluated or approximated via finite difference as Eq.

Original: As Eq.
Revised: As Eq.

Original: 18 already solves the first term on the right hand side of Eq.
Revised: 18 already solves the first term on the right hand side of Eq.

Original: 16, the remaining issue is d L ( w k , y K ) /d
Revised: 16, the remaining issue is d L ( a k , b K ) /d

Original: y k ′ .
Revised: b k ′ .

Original: To solve this term, we expand it recursively as Eq. 19 and take Eq.into it.
Revised: To solve this term, we expand it recursively as Eq. 19 and take Eq.into it.

Original: [Equation]
Revised: [Equation]

Original: And the above solving process can be described by the procedure grad-2-level( x , w k ) of Alg. 1.
Revised: And the above solving process can be described by the procedure grad-2-level( x , a k ) of Alg. 1.

Original: Specifically, the iterative update of ←−−− y k ′ +1 in line 15 corresponds to recursively expanding Eq.with Eq. 17, and the iterative update of ←− w in line 14 corresponds to recursively expanding 
Revised: Specifically, the iterative update of ←−−− b k ′ +1 in line 15 corresponds to recursively expanding Eq. 19 with Eq. 17, and the iterative update of ←− a in line 14 corresponds to recursively expanding Eq. 16 with

Original: Eq.
Revised: Eq.

Original: 16 with Eq.

Original: 18 and Eq. 19.
Revised: 18 and Eq. 19.

Original: Upon the return of grad-2-level( x , w k ) of Alg. 1, we have ←− w = d L ( w k , y K ) /d
Revised: Upon the return of grad-2-level( x , a k ) of Alg. 1, we have ←− a = d L ( a k , b K ) /d

Original: w k .
Revised: b k .

Original: The complexity of the Hessian-vector product in line 14 and 15 of Alg.
Revised: The complexity of the Hessian-vector product in line 14 and 15 of Alg.

Original: 1 may be reduced using finite difference following (Domke, 2012) as Eq. 20.
Revised: 1 may be reduced using finite difference following (Domke, 2012) as Eq. 20.

Original: [Equation]
Revised: [Equation]

Original: Theorem 2.
Revised: Theorem 2.

Original: After the procedure grad-dag( x , y k 0 0 , ..., y k i i ) in Alg.
Revised: After the procedure grad-dag( x , a k 0 0 , ..., a k i i ) in Alg.

Original: 2 executes, we have the return value
Revised: 2 executes, we have the return value

Original: d L ( y k 0 0 , ..., y k i i , y K>i ) /d y k i i =
Revised: d L ( a k 0 0 , ..., a k i i , a K>i ) /d a k i i =

Original: ←− y i .
Revised: ←− a i .

Original: Proof.
Revised: Proof.

Original: Consider computing the target gradient with DAG G .
Revised: Consider computing the target gradient with DAG G .

Original: The y ki ’s gradient is composed of its own contribution to L in addition to the gradient from its children y j ∈ C ( y i ) .
Revised: The a ki ’s gradient is composed of its own contribution to L in addition to the gradient from its children a j ∈ C ( a i ) .

Original: Further, as we are considering the optimized children y Kj , we expand the children node y j as Fig. 4.
Revised: Further, as we are considering the optimized children a Kj , we expand the children node a j as Fig. 4.

Original: Then, we have:
Revised: Then, we have:

Original: [Equation]
Revised: [Equation]

Original: The first term on the right-hand side of Eq.
Revised: The first term on the right-hand side of Eq.

Original: 21 can be trivially evaluated.
Revised: 21 can be trivially evaluated.

Original: The ∂ y k j j /∂ y k
Revised: The ∂ a k j j /∂ a k

Original: i
Revised: i

Original: i can be evaluated as Eq.
Revised: i can be evaluated as Eq.

Original: And the d L ( y k 0 0 , ..., y k j − 1 j − 1 , y K ≥ j ) /d y k j j can be iteratively expanded as Eq. 
Revised: And the d L ( a k 0 0 , ..., a k j − 1 j − 1 , a K ≥ j ) /d a k j j can be iteratively expanded as Eq. 19.

Original: We highlight several key differences between Alg. 2 and Alg. 1 which are reflected in the implementation of Alg. 2:
Revised: We highlight several key differences between Alg. 2 and Alg. 1 which are reflected in the implementation of Alg. 2:

Original: • The gradient evaluation of current node y i requires gradient of its plural direct children y j ∈ C ( y i ) , instead of the single child in 2-level case.
Revised: • The gradient evaluation of current node y i requires gradient of its plural direct children a j ∈ C ( a i ) , instead of the single child in 2-level case.

Original: The children traversal part of Eq.corresponds to the two extra for loop in line 8 and 14 of Alg. 2.
Revised: The children traversal part of Eq.corresponds to the two extra for loop in line 8 and 14 of Alg. 2.

Original: • The gradient ascent update of child latent parametery k
Revised: • The gradient ascent update of child latent parameter

Original:  j +

Revised: a k j +

Original: j
Revised: j

Original: [Equation] can be conducted trivially only if C ( y j ) is empty, otherwise the gradient has to be evaluated recursively using Eq. 21.
Revised: ← a k j j + αd L ( a k 0 0 , ..., a k j j , a K>j ) /d a k j j can be conducted trivially only if C ( a j ) is empty, otherwise the gradient has to be evaluated recursively using Eq. 21.

Original: And this part corresponds to the recursive call in line 11 of Alg. 2.
Revised: And this part corresponds to the recursive call in line 11 of Alg. 2.

Original: And the other part of Alg. 2 is the same as Alg. 1.
Revised: And the other part of Alg. 2 is the same as Alg. 1.

Original: So the rest of the proof follows Thm. 1.
Revised: So the rest of the proof follows Thm. 1.

Original: Similarly, the Hessian-vector product in line 17 and 18 of Alg.
Revised: Similarly, the Hessian-vector product in line 17 and 18 of Alg.

Original: 2 may be approximated as Eq.
Revised: 2 may be approximated as Eq.

Original: However, this does not save Alg. 2 from an overall complexity of Θ( K N ) .
Revised: However, this does not save Alg. 2 from an overall complexity of Θ( K N ) .

Original: A. T HE C OMPLETE F ORMULA FOR S EC . 3.1 AND S EC . 3. 
Revised: A. T HE C OMPLETE F ORMULA FOR S EC . 3.1 AND S EC . 3.

Original: In this section, we provide the complete formula on y i related gradient for Sec.
Revised: In this section, we provide the complete formula on y i related gradient for Sec.

Original: 3.1 and Sec. 3.2.
Revised: 3.1 and Sec. 3.2.

Original: Specifically, Eq. 22 is paired with Eq. 10, and Eq.
Revised: Specifically, Eq. 22 is paired with Eq. 10, and Eq.

Original: 23 is paired with Eq.
Revised: 23 is paired with Eq.

Original: d L ( w 1: T , y 1: T )
Revised: d L ( w 1: T , y 1: T )

Original: [Equation]
Revised: [Equation]

Original: d 
Revised: d L

Original: [Equation]
Revised: [Equation]

Original: j
Revised: y

Original: [Equation]
Revised: [Equation]

Original:  (cid:125) 
Revised: j ) [Equation] (cid:123)(cid:122) (cid:125) ignored

Original: [Table]
Revised: [Table]

Revised: BAO

Original: [Equation]
Revised: [Equation]

Revised: by

Revised: [Equation]

Original:  where w 0 >
Revised: , where w 0 >

Original: i , y 0 >
Revised: i , y 0 >i =

Revised: f ( x , w k 1 1 , ..., w k

Original: i
Revised: i

Original: [Equation] x , 
Revised: i , y k ′ 1 1 , ...,

Original: [Equation] [Equation]
Revised: y k ′ i i ) (23)

Original: A. A N E XAMPLE OF E XECUTION OF A LG .
Revised: A. A N E XAMPLE OF E XECUTION OF A LG .

Original: In this section, we provide an example of the full percedure of execution of Alg. 2 in Fig.
Revised: In this section, we provide an example of the full procedure of execution of Alg. 2 in Fig.

Original: The setup is as Fig.
Revised: The setup is as Fig.

Original: 5.(0): we have N = 3 latent y 1 , y 2 , y 3 and gradient ascent step K = 2 , connected by a DAG shown in the figure.
Revised: 5.(0): we have N = 3 latent a 1 , a 2 , a 3 and gradient ascent step K = 2 , connected by a DAG shown in the figure.

Original: A. M ORE I MPLEMENTATION D ETAILS
Revised: A. E XTENDING THE A NALYSIS S EC . 3 TO GENERAL DAG CASE

Revised: As the Alg. 2 and Alg. 4 are applicable to general SAVI (Kim et al., 2018; Marino et al., 2018) beyond bit allocation, it is helpful to understand their merit to extend the analysis in Sec. 3 from bit allocation to general DAG scenario. In this section, we consider the same problem setup as Sec. 4.2.

Revised: Similar to bit allocation case, BAO has the gradient incomplete and gradient value incorrect problem.

Revised: The gradient incomplete issue is presented as Eq. 24, and gradient value incorrect issue is presented as Eq.

Revised: [Equation]

Revised: ignored by BAO

Revised: [Equation]

Revised: [Equation]

Revised: approximation

Revised: [Equation]

Revised: A. M ORE I MPLEMENTATION D ETAILS

Original: In the main text, we use y i as all the latent variable related to residual.
Revised: In the main text, we use y i as all the latent variable related to residual.

Original: In practice, it is divided into y i , z i , ∆ yi , which refer to the first level latent of residual, second level latent of residual and quantization step size of first level latent of residual respectively.
Revised: In practice, it is divided into y i , z i , ∆ y i , which refer to the first level latent of residual, second level latent of residual and quantization step size of first level latent of residual respectively.

Original: In practice, as BAO (Xu et al., 2022), all of the 3 parts are involved in SAVI jointly.
Revised: In practice, as BAO (Xu et al., 2022), all of the 3 parts are involved in SAVI jointly.

Original: We note that this is not a problem as they fully factorize.
Revised: We note that this is not a problem as they fully factorize.

Original: And for DVC (Lu et al., 2019), w i indeed represent the latent of motion.
Revised: And for DVC (Lu et al., 2019), w i indeed represent the latent of motion.

Original: As for
Revised: As for

Original: DVC, the motion has only one level of latent.
Revised: DVC, the motion has only one level of latent.

Original: However for DCVC (Li et al., 2021), w i is divided into w i , v i , ∆ wi , which refer to the first level latent of motion, second level latent of motion and quantization step size of first level latent of motion respectively.
Revised: However for DCVC (Li et al., 2021), w i is divided into w i , v i , ∆ wi , which refer to the first level latent of motion, second level latent of motion and quantization step size of first level latent of motion respectively.

Original: Similar to y i , all of the 3 parts are involved in SAVI jointly, and this is not a problem as they fully factorize.
Revised: Similar to y i , all of the 3 parts are involved in SAVI jointly, and this is not a problem as they fully factorize.

Original: Following BAO (Xu et al., 2022), we set the target λ 0 = { 256 , 512 , 1024 , 2048 } , which also follows the baselines (Lu et al., 2019; Li et al., 2021).
Revised: Following BAO (Xu et al., 2022), we set the target λ 0 = { 256 , 512 , 1024 , 2048 } , which also follows the baselines (Lu et al., 2019; Li et al., 2021).

Original: We adopt the official pre-train models for both of the baseline methods (Lu et al., 2019; Li et al., 2021).
Revised: We adopt the official pre-train models for both of the baseline methods (Lu et al., 2019; Li et al., 2021).

Original: We do not have a training dataset or implementation details for training amortized encoder / decoder as all the experiments are performed on official pre-trained models.
Revised: We do not have a training dataset or implementation details for training amortized encoder / decoder as all the experiments are performed on official pre-trained models.

Original: For gradient ascent, we set K = 2000 for the first I frame and
Revised: For gradient ascent, we set K = 2000 for the first I frame and

Original: K = 400 for all other P frames.
Revised: K = 400 for all other P frames.

Original: On average, the gradient ascent steps for each frame is 920 , which is smaller than 2000 in BAO.
Revised: On average, the gradient ascent steps for each frame is 920 , which is smaller than 2000 in BAO.

Original: A. M ORE Q UANTITATIVE R ESULTS
Revised: A. M ORE Q UANTITATIVE R ESULTS

Original: In this section we present more quantitative results.
Revised: In this section we present more quantitative results.

Original: In Tab. 4 we show the BD-PSNR of our proposed method and other methods as a supplementary to the BD-BR results (Tab. 1).
Revised: In Tab. 4 we show the BD-PSNR of our proposed method and other methods as a supplementary to the BD-BR results (Tab. 1).

Original: Furthermore, in Fig.
Revised: Furthermore, in Fig.

Original: 6, we present R-D curve on all classes of HEVC CTC and UVG dataset as a supplementary to the
Revised: 6, we present R-D curve on all classes of HEVC CTC and UVG dataset as a supplementary to the

Original: HEVC Class D plot (Fig. 2). 
Revised: HEVC Class D plot (Fig. 2). [Table] Cls

Original: [Table]
Revised: [Table]

Revised: Cls

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: BAO (

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: Proposed

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Original: [Figure]
Revised: [Figure]

Revised: A. C OMPLEXITY & S CALABILITY

Revised: We perform additional evaluation to compare the proposed method with BAO (Xu et al., 2022) in terms of temporal complexity and memory cost.

Revised: The evaluation result can be found in Fig.

Revised: The general result is that our approach is ≈ 2 .

Revised: 8 times slower and cost ≈ 2 .

Revised: 0 times memory than BAO, despite the optimization stepsize is smaller.

Revised: This extra complexity comes from the cost of sequential optimization of latent.

Revised: And our current method in its na ¨ ıve form is slower than BAO while performs better.

Revised: Jointly consider RD performance, time and memory, our method does not dominate BAO.

Revised: However, as our approach enables a sequential style semi-amortized variational inference (SAVI) (Kim et al., 2018; Marino et al., 2018) on latents, there exists a very simple trick to speed it up.

Revised: Moreover, this trick also resolves the scalability issue.

Revised: Specifically, to optimize the i th frame’s latent, we do not compute the R-D cost of all the frames after it as we do now.

Revised: Instead, we limit the

Revised: R-D cost computation to a small fixed size of frames.

Revised: Formally, we approximate the gradient as: d L ( w 1: T , y 1: T )

Revised: d w i

Revised: ≈ i + C (cid:88)

Revised: j = i

Revised: d L j ( w 1: j , y 1: j ) d w i , d L ( w 1: T , y 1: T ) d y i

Revised: =

Revised: i + C (cid:88)

Revised: j = i

Revised: d L j ( w 1: j , y 1: j ) d y i (26) , where C is a preset constant indicating the number of future frames we included for consideration.

Revised: With this trick, our algorithm approach cost only ≈ 50% of time and ≈ 60% of memory compared with BAO, while remains a superior performance ( ≈ 5% better in BDBR)

Revised: (Ours (fast) in Fig.

Revised: 7, the results are based on DVC Class D c = 2 ).

Revised: With this trick, jointly consider RD performance, time and memory, our approach clearly dominates BAO.

Revised: Furthermore, with this trick, the scalability issue of our approach is significantly ellivated.

Revised: As shown in Fig.

Revised: 8, the memory cost our approach with this trick is constant to GoP size, while that of BAO and our approach without this trick grows linearly with GoP size.

Revised: This means that with this trick, our approach becomes scalable to

Revised: GoP, is superior than BAO

Revised: A. I MPACT ON OEU

Revised: Another interesting question to ask is whether the sequential updating algorithm (Alg. 4) benefits the OEU (Lu et al., 2020).

Revised: Indeed, OEU (Lu et al., 2020) and BAO (Xu et al., 2022) are quite similar at the first glance.

Revised: However, it is important to note that the theoretical foundation of BAO and this paper is SAVI (Kim et al., 2018; Marino et al., 2018).

Revised: However, OEU does not fit into SAVI framework.

Revised: More specifically, its encoder parameter to be updated does not factorizes as the DAG defined by variational posterior.

Revised: Thus, applying Alg. 4 is incorrect.

Revised: To verify this empirically, we change the OEU from BAO’s joint optimization to ours sequential optimization (Alg. 4), and the results show that this change degrades R-D performance (See COEU line in Fig. 9).

Original: A. M ORE A NALYSIS
Revised: A. M ORE A NALYSIS

Original: In this section, we extend the analysis on why the proposed approach works and what is the difference between the proposed approach and BAO (Xu et al., 2022).
Revised: In this section, we extend the analysis on why the proposed approach works and what is the difference between the proposed approach and BAO (Xu et al., 2022).

Original: In the approximate SAVI on DAG latent (Alg. 4), we solve SAVI approximately latent by latent in topological order.
Revised: In the approximate SAVI on DAG latent (Alg. 4), we solve SAVI approximately latent by latent in topological order.

Original: For bit allocation of NVC with 10 frames, this topological order is y 0 , w 1 , y 1 , ..., w 9 , y 9 , where y 0 is the latent of I frame, w i is the motion latent of i th P frame and y i is the residual latent of i th P frame.
Revised: For bit allocation of NVC with 10 frames, this topological order is y 0 , w 1 , y 1 , ..., w 9 , y 9 , where y 0 is the latent of I frame, w i is the motion latent of i th P frame and y i is the residual latent of i th P frame.

Original: In Fig.
Revised: In Fig.

Original: 7, we show the relationship between R-D cost and the stage of approximate SAVI.
Revised: 10, we show the relationship between R-D cost and the stage of approximate SAVI.

Original: We can see that the R-D cost reduces almost consistently with the growing of SAVI stage, which indicates that our approximate SAVI on DAG (Alg. 4) is successful.
Revised: We can see that the R-D cost reduces almost consistently with the growing of SAVI stage, which indicates that our approximate SAVI on DAG (Alg. 4) is successful.

Original: Specifically, despite our approach is inferior to
Revised: Specifically, despite our approach is inferior to

Original: BAO (Xu et al., 2022) upon the convergence of y 3 , it attains significant advantage over BAO after y 9 converges.
Revised: BAO (Xu et al., 2022) upon the convergence of y 3 , it attains significant advantage over BAO after y 9 converges.

Original: In Fig.
Revised: In Fig.

Original: 8, we compare the distribution of R-D cost, PSNR and Bpp across frame and latent of the baseline DVC (Lu et al., 2019), BAO Xu et al.
Revised: 11, we compare the distribution of R-D cost, PSNR and Bpp across frame and latent of the baseline DVC (Lu et al., 2019), BAO Xu et al.

Original: and the proposed approach.
Revised: and the proposed approach.

Original: For R-D cost, it is obvious that our proposed approach’s R-D cost is lower than BAO and baseline, which indicates a better R-D performance.
Revised: For R-D cost, it is obvious that our proposed approach’s R-D cost is lower than BAO and baseline, which indicates a better R-D performance.

Original: For bpp, it is interesting to observe that despite all three methods have similar bpp of motion related latent w 1: T , the bpp of residual related latent y 1: T is quite different.
Revised: For bpp, it is interesting to observe that despite all three methods have similar bpp of motion related latent w 1: T , the bpp of residual related latent y 1: T is quite different.

Original: Specifically, BAO increases the bpp of y 1: T compared with baseline, while our approach decreases the bpp of y 1: T compared with baseline.
Revised: Specifically, BAO increases the bpp of y 1: T compared with baseline, while our approach decreases the bpp of y 1: T compared with baseline.

Original: This explains why our approach has lower bitrate compared with BAO, and also explains why our approach has significantly less bitrate error.
Revised: This explains why our approach has lower bitrate compared with BAO, and also explains why our approach has significantly less bitrate error.

Original: For the PSNR metric, both our approach and BAO significantly improve the baseline.
Revised: For the PSNR metric, both our approach and BAO significantly improve the baseline.

Original: And the difference between proposed approach and BAO is not obvious.
Revised: And the difference between proposed approach and BAO is not obvious.

Original: We can conclude that the benefits of the proposed approach over BAO comes from the bitrate saving instead of quality enhancing.
Revised: We can conclude that the benefits of the proposed approach over BAO comes from the bitrate saving instead of quality enhancing.

Original: A. Q UALITATIVE R ESULTS
Revised: A. Q UALITATIVE R ESULTS

Original: In Fig.
Revised: In Fig.

Original: 9, Fig. 10, Fig.
Revised: 12, Fig. 13, Fig.

Original: 11 and Fig.
Revised: 14 and Fig.

Original: 12, we present the qualitative result of our approach compared with the baseline approach.
Revised: 15, we present the qualitative result of our approach compared with the baseline approach.

Original: We note that compared with the reconstruction frame of baseline approach, the reconstruction frame of our proposed approach preserves significantly more details with lower bitrate, and looks much more similar to the original frame.
Revised: We note that compared with the reconstruction frame of baseline approach, the reconstruction frame of our proposed approach preserves significantly more details with lower bitrate, and looks much more similar to the original frame.

Original: We intentionally omit the qualitative comparison with BAO (Xu et al., 2022) as it is not quite informative.
Revised: We intentionally omit the qualitative comparison with BAO (Xu et al., 2022) as it is not quite informative.

Original: Specifically, from Fig.
Revised: Specifically, from Fig.

Original: 2 we can observe that the PSNR difference of BAO and our approach is very small (within ± 0 . 1 dB).
Revised: 2 we can observe that the PSNR difference of BAO and our approach is very small (within ± 0 . 1 dB).

Original: And our main advantage over BAO comes from bitrate saving instead of quality improvement.
Revised: And our main advantage over BAO comes from bitrate saving instead of quality improvement.

Original: Thus the qualitative difference between the proposed method and BAO is likely to fall below just noticeable difference (JND).
Revised: Thus the qualitative difference between the proposed method and BAO is likely to fall below just noticeable difference (JND).

Original: A. M ORE D ISCUSSION
Revised: A. M ORE D ISCUSSION

Original: Other weakness includes scalability.
Revised: Other weakness includes scalability.

Original: Our method requires jointly considering all the frame inside the GoP, which is impossible when the GoP size is large or when GoP size is unknown for live streaming tasks.
Revised: Our method requires jointly considering all the frame inside the GoP, which is impossible when the GoP size is large or when GoP size is unknown for live streaming tasks.

Original: Furthermore, currently the gradient ascent step number is merely chosen as an empirical sweet spot between speed and performance.
Revised: Furthermore, currently the gradient ascent step number is merely chosen as an empirical sweet spot between speed and performance.

Original: A thorough grid search is desired to better understand its effect on performance.
Revised: A thorough grid search is desired to better understand its effect on performance.