Paper: FFCA-Net: Stereo Image Compression via Fast Cascade Alignment of Side Information (ICLR-2024)

Reviewer comment (Novelty):
The foundational concepts of the paper, including the coarse-to-fine cascade structure, hourglass-like stereo rectification, and feature fusion layer, appear to be somewhat straightforward, potentially raising questions about the novelty of the approach.

Author response:
We truly understand your concerns, but respectfully, we disagree with that assertion. To the best of our knowledge, the coarse-to-fine cascade architecture we propose, along with its constituent modules, represents a novel application in the field of image compression. Compared to previous works, our approach offers a more rational integration of priors and a lighter, faster feature-matching architecture. 

- For example, in [1], there was an attempt to use stereo-matching techniques for feature domain matching. However, the disparity estimation module was found to be overly complex, leading to significant differences in inference speed compared to our proposed solution (relevant experimental results can be found in Section 4.2).
- In addition, it is extremely difficult to adapt many established stereo matching techniques to the feature domain. For example, GC-Net [1] requires about 0.9 seconds to infer a single image on KITTI using a GPU [2]. If we were to adapt it to the feature domain, the computational cost would significantly increase due to the considerable computational overhead introduced by multiple channels. In contrast, our proposed "hourglass-based sparse stereo refinement" achieves an inference time of just 0.02 seconds on KITTI.

- Unlike complex parameterized neural networks, our cascaded structure enables model-level editing, such as replacing coarse or fine alignment modules, making it convenient for future work.


[1]Kendall A, Martirosyan H, Dasgupta S, et al. End-to-end learning of geometry and context for deep stereo regression[C]//Proceedings of the IEEE international conference on computer vision. 2017: 66-75.

[2]Tankovich V, Hane C, Zhang Y, et al. Hitnet: Hierarchical iterative tile refinement network for real-time stereo matching[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021: 14362-14372.

Reviewer comment (Writing):
Typos and Formatting: On Page 3, change “simultaneously” to “Simultaneously.” Ensure consistent punctuation at the end of equations (1-6, 8-9) with periods, and a comma at the end of equation (7).

Author response:
We sincerely appreciate your valuable feedback on our work and the constructive suggestions you have provided. We apologize for any misunderstandings caused by printing errors. In the revised version, we have corrected these errors and thoroughly reviewed the entire article to prevent similar mistakes from happening again. We hope this time it will offer you a better reading experience!

Reviewer comment (Theory):
The improvement of network encoding speed in this manuscript mainly relies on prior knowledge of binocular stereo images. However, these prior assumptions may not be applicable in all cases. Overreliance on prior knowledge may result in the loss of information, which the neural network itself can learn that is beneficial for encoding reconstruction.

Author response:
Thank you for bringing up your concerns in the weaknesses section. We have thoroughly reviewed and addressed your worries. Please let us offer suitable explanations for these concerns.

**W1**: The prior assumptions may not be applicable in all cases. Overreliance on prior knowledge may result in the loss of information, which the neural network itself can learn that is beneficial for encoding reconstruction.

**Rw1**: Thank you for your constructive insight! Although the prior algorithm we provided may not be effective in certain non-stereo image compression scenarios, we believe that even in these situations, our work still offers several insightful aspects.

-  Utilizing these non-learning priors can effectively accelerate the inference process of the model. Generally, employing complex parameterized neural networks to learn priors often leads to unsatisfactory encoding and decoding speeds.
- While the priors may vary across different scenarios, our cascaded structure provides a valuable framework and solution for integrating this prior knowledge effectively.

However, it is acknowledged that the applicability of our approach may be limited in certain scenarios. In our future works, we will strive to address and potentially alter this limitation.

Reviewer comment (Reproducibility):
What is the relationship between Main image and Side image? Is the side image encoded? How to encode? How to measure the final decoding time?

Author response:
**Q1**: What is the relationship between Main image and Side image? Is the side image encoded? How to encode? How to measure the final decoding time?

**R1**: Thank you sincerely for your feedback! We deeply apologize for any confusion our description may have caused you. Please allow me to provide further clarification.

- In the information theory [1], the terms "side/main information" are proprietary terminology within the DSC framework, which proves that compression of two correlated sources that do not communicate with each other could achieve compression efficiency with mutual communication. Since our work also falls within the DSC framework, we have used the same terminology [2] [3] in our work, **referring to different perspectives in stereoscopic imagery as "main/side image."**
- **The side image is also compressed**. In practical applications, the concept of "main/side image" is relative. Users can choose their current perspective as the main image and the others as the side image for auxiliary decoding, and vice versa. This means that after another round of encoding and decoding, the contents of the main and side images can be swapped, and the difference can be adjusted to decode the other perspective.
-  Since this process is symmetric, the side image is encoded as the main one, independently. **the average decoding time for both perspectives is theoretically equivalent to decoding a single image**. Hence, in our experiments, we simulate the decoding time for a pair of stereo images by using twice the time it takes to decode a single image.

Thank you once again for your feedback. We acknowledge that our previous submission lacked clarity in explaining this issue. We will make sure to provide additional clarification in the revised version.



[1] Cover, T. 1975. A proof of the data compression theorem of Slepian and Wolf for ergodic sources (Corresp.). *IEEE* *Transactions on Information Theory*, 21(2): 226–228.

[2] Huang Y, Chen B, Qin S, et al. Learned distributed image compression with multi-scale patch matching in feature domain[C]//Proceedings of the AAAI Conference on Artificial Intelligence. 2023, 37(4): 4322-4329.

[3] Ayzik S, Avidan S. Deep image compression using decoder side information[C]//Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XVII 16. Springer International Publishing, 2020: 699-714.

Reviewer comment (Experiments):
Which module mentioned in this paper has the most significant acceleration on image decoding?

Author response:
**Q2:** Which module mentioned in this paper has the most significant acceleration on image decoding?

**R2:** We apologize for the lack of comprehensive experimental results on acceleration in the initial submission. Please allow me to present the following ablation result to address your concerns.

We have divided our proposed solution into three parts: coarse-grained stereo patch matching, fine-grained hourglass-based sparse stereo refinement, and feature fusion using the FFF module. To ensure fairness, we used MSFDPM [1] as our baseline to show the most significant module for acceleration on image decoding.

- **Stereo Patch Matching**:  We have selected Multi-Scale Patch Matching [1] as the baseline method for comparison. Both approaches were evaluated on a feature layer of size $128\times256\times256$.

  |  Input Size$\downarrow$   | Method$\downarrow$, Speed$\rightarrow$ |     CPU     |    GPU     |
  | :-----------------------: | :------------------------------------: | :---------: | :--------: |
  | 128$\times$256$\times$256 |    Stereo Patch-Matching (Proposed)    | **218.3ms** | **11.8ms** |
  | 128$\times$256$\times$256 |  Multi-scale Patch-Matching (MSFDPM )  |  2163.3ms   |  138.8ms   |

  ​	It is evident from the results that our algorithm outperforms the baseline method by nearly tenfold, both in CPU and GPU environments. This   significant speed improvement is attributed to our efficient parallel computing techniques, which have proven to be reliable.

- **Fast Feature Fusion**: We selected the Feature Fusion module [1] as the baseline for comparison. Our proposed FFF  module is an enhanced version of the Feature Fusion module, with a smaller parameter count and faster inference speed. We evaluated the inference speed on CPU and the model parameter count for both methods, with an input size of $3\times512\times512$, as shown in the table below. The results confirm the effectiveness of our model.

  | Input Size$\downarrow$  | Method$\downarrow$, Efficiency$\rightarrow$ | Speed(CPU)  | Parameters |
  | :---------------------: | :-----------------------------------------: | :---------: | :--------: |
  | 3$\times$512$\times$512 |       Fast Feature Fusion (Proposed)        | **577.5ms** | **3.04M**  |
  | 3$\times$512$\times$512 |           Feature Fusion (MSFDPM)           |   683.9ms   |   7.02M    |



Based on our ablation experiments and in comparison to the baseline MSDFPM, **the most significant speed improvement is observed in our proposed stereo patch matching technique.** This high-speed and robust coarse-grained alignment provides a solid foundation for subsequent decoding processes. 

Furthermore, our proposed module 'Hourglass-based Sparse Stereo Refinement' performs fine-grained adjustment on top of coarse-grained matching, which is not present in MSFDPM. However, even with the addition of pixel-level correction, our method still exhibits a significant speed advantage. As shown in Section 4.2 of the original paper, the decoding speed of our approach is 1/5 that of MSFDPM (4.9s/23.85s). 

As far as we know, previous works utilizing the DSC framework did not integrate stereo matching techniques. To further confirm the contribution of our fine-grained module in terms of acceleration, we chose the Parametric Skip Function [2], a crucial component of a non-DSC stereo image compression algorithm (DSIC), as the baseline method. This method aims to predict complete inter-feature disparities using stacked residual blocks. We assessed the inference speed on CPU and the model parameter count for both methods, with an input size of $3\times512\times512$, as indicated in the table below:

| Input Size$\downarrow$  | Method$\downarrow$, Efficiency$\rightarrow$ | Speed(CPU) | Parameters |
| :---------------------: | :-----------------------------------------: | :--------: | :--------: |
| 3$\times$512$\times$512 |  Hourglass-based SS Refinement (Proposed)   | **308ms**  | **0.24M**  |
| 3$\times$512$\times$512 |       Parametric Skip Fuction (DSIC)        |  1194.3ms  |   8.64M    |

In comparison to the baseline, our approach exhibits a significant speed improvement, being only 1/4 of the baseline's runtime. Additionally, our method achieves a parameter count that is merely 1/40 of the baseline's parameter count.

We express our gratitude once again for your insightful commentary. We shall incorporate these experiments into the appendix, and you will find them in Section 6.2 of the revised version.

[1] Huang Y, Chen B, Qin S, et al. Learned distributed image compression with multi-scale patch matching in feature domain[C]//Proceedings of the AAAI Conference on Artificial Intelligence. 2023, 37(4): 4322-4329.

[2] Liu J, Wang S, Urtasun R. Dsic: Deep stereo image compression[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision. 2019: 3136-3145.

Reviewer comment (Reproducibility):
Does the bpp term in the loss function only contain the potential representation z, without using any prior knowledge from other learning based compression methods? Is it already included? Are there any special considerations for model optimization based on MS-SSIM?

Author response:
**Q5**:  Does the bpp term in the loss function only contain the potential representation z, without using any prior knowledge from other learning-based compression methods? Is it already included? Are there any special considerations for model optimization based on MS-SSIM?

**R5**: Thank you for posing this inquiry. We regret that we were unable to articulate this matter clearly in the submitted version.

**Regarding the 'Does the bpp term in the loss function only contain the potential representation z?',** the bpp term typically encompasses prior knowledge and is determined by the components of the baseline single-image encoder-decoder we have employed. In this manuscript, we utilized [1] as the baseline single-image compression model, thereby incorporating the mixture Gaussian hyperprior offered in [1] within the bpp term, as depicted in the following equation.
$$
\begin{aligned}
\mathcal{R}({\boldsymbol{z}})= \mathcal{R}(\hat{\boldsymbol{z}})+\mathcal{R}(\hat{\boldsymbol{u}}) 
\end{aligned}
$$

$$
\mathcal{R}(\hat{\boldsymbol{z}})=\mathbb{E}\left[-\log \left(p_{\hat{\boldsymbol{z}} \mid \hat{\boldsymbol{u}}}(\hat{\boldsymbol{z}} \mid \hat{\boldsymbol{u}})\right)\right]
$$

$$
\mathcal{R}(\hat{\boldsymbol{u}}) =\mathbb{E}\left[-\log \left(p_{\hat{\boldsymbol{u}} \mid \boldsymbol{\psi}}(\hat{\boldsymbol{u}} \mid \boldsymbol{\psi})\right)\right]
$$

$$
\text{where} \quad p_{\hat{\boldsymbol{z}} \mid \hat{\boldsymbol{u}}}(\hat{\boldsymbol{z}} \mid \hat{\boldsymbol{u}}) \sim \sum_{k=1}^K \boldsymbol{w}^{(k)} \mathcal{N}\left(\boldsymbol{\mu}^{(k)}, \boldsymbol{\sigma}^{2(k)}\right)
$$

To provide readers with a better understanding of this aspect, we will include these explanatory details in the revised appendix.

**Regarding the " Are there any special considerations for model optimization based on MS-SSIM?",** In practice, it is common to optimize using MS-SSIM as a reconstruction loss. Although this may lead to a slight decrease in PSNR during testing, we have found in our experiments that using MSE as the reconstruction loss can sometimes produce unsatisfactory results. For example, at low bit rates, models trained with MSE loss (such as BPG) may have acceptable PSNR but often suffer from distortions and artifacts around object edges (this point is also illustrated in the original submission through visualizations). On the other hand, using MS-SSIM helps address these issues. We believe that image tearing and distortions are more unacceptable than a slightly blurred visual effect, which is a significant reason for choosing MS-SSIM as the reconstruction loss.



[1] Cheng Z, Sun H, Takeuchi M, et al. Learned image compression with discretized gaussian mixture likelihoods and attention modules[C]//Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2020: 7939-7948.

[2] Huang Y, Chen B, Qin S, et al. Learned distributed image compression with multi-scale patch matching in feature domain[C]//Proceedings of the AAAI Conference on Artificial Intelligence. 2023, 37(4): 4322-4329.

[3] Ayzik S, Avidan S. Deep image compression using decoder side information[C]//Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XVII 16. Springer International Publishing, 2020: 699-714.

[4] Zhang X, Shao J, Zhang J. LDMIC: Learning-based Distributed Multi-view Image Coding[C]//The Eleventh International Conference on Learning Representations. 2022.

Reviewer comment (Experiments):
In the ablation experiment, the Fast Feature Fusion module slightly reduced PSNR. Which result proves that the FFF module can achieve faster decoding? How many iterations can the FFF module undergo to achieve the best results?

Author response:
**Q6:** In the ablation experiment, the Fast Feature Fusion module slightly reduced PSNR. Which result proves that the FFF module can achieve faster decoding? How many iterations can the FFF module undergo to achieve the best results?

**R6**: We appreciate your constructive comments! Please allow me to address each of your questions separately.

- **Which result proves that the FFF module can achieve faster decoding?**  R:  Thank you for pointing out this aspect. You can find the additional experimental results in Table of Respond 2. The ablation experiments demonstrate that our FFF module is more lightweight (3.04M/7.02M) and exhibits improved inference speed (577.5ms/683.9ms). We have also included the corresponding experimental results in the revised version, which can be found in Section 6.2.

- **How many iterations can the FFF module undergo to achieve the best results?** R: We apologize for any misunderstanding caused by the omission in the description. The number of iterations in the FFF module is indeed linked to the number of scales of the multi-scale features separated by the baseline encoder. For instance, in this work, we employed a baseline encoder-decoder that decouples the image into four distinct scale feature layers, denoted as .$h^1_\hat{x},h^2_\hat{x},h^3_\hat{x},h^4_\hat{x}$, arranged in decending order of scale. Each iteration of the FFF module can be abstracted by the following equation.
  $$
  \phi^i=FFF^i(\phi^{i+1},h^i_\hat{x},h^i_{y^{\star\star}}) \quad i=1,2,3,4
  $$
  Here, $FFF^i$ refers to the FFF module at the i-th iteration, and $h^i_{y^{\star\star}}$ represents the side information obtained through coarse to fine matching. As observed, it takes **4 iterations** of FFF to fuse information from all four feature scales. 

Additionally, omitting any scale of features in the decoding stage hinders achieving improved image reconstruction[1]. Here, we conduct an ablation study using the InStero2k dataset to illustrate this point. The variable in focus is the scale of features used by the decoder, with $\lambda=0.01$.

  | The fused features | $h^{1,2,3,4}$ | $h^{1,2,3}$ | $h^{1,2}$ | $h^1$ |
  | :----------------: | :---------------: | :-----------: | :-------: | :---: |
  |      **PSNR**      |       32.8        |     32.1      |   30.3    | 29.2  |

This aligns with our expectations, as the **4 iterations of FFF can effectively fuse features from all scales and achieve optimal reconstruction results.** Once again, we appreciate your feedback, and we will include these supplementary explanations in Section 6.2 of the revised version.