Paper: Dual Associated Encoder for Face Restoration (ICLR-2024)

Reviewer comment (Writing):
The full name of the proposed framework, DAEFR, is missing. It should be mentioned on its first occurrence in the paper.

Author response:
The acronym 'DAEFR' stands for the title of our paper, which is an initialism of "**D**ual **A**ssociated **E**ncoder for **F**ace **R**estoration."
We will revise these parts in our updated manuscript.

Reviewer comment (Writing):
Check the spellings. For example, “recently” on the beginning of second paragraph in the “Vector Quantized Codebook Prior” of the related work.

Author response:
We will revise these parts in our updated manuscript.

Reviewer comment (Novelty):
The contribution looks marginal to me since all the methods used in different stage are well designed and demonstrated. Adding another stream for low-resolution might not be a major contribution for a top-tier venue like ICLR.

Author response:
**[Q1] The contribution looks marginal to me since all the methods used in the different stages are well-designed and demonstrated. Adding another stream for low-resolution might not be a major contribution for a top-tier venue like ICLR.**

We would like to further clarify the novelty and effectiveness of this work by putting this work in proper context and emphasizing the empirical results over state-of-the-art approaches. 

* We would like to clarify that although some proposed model components are similar to prior arts, it requires **meticulous algorithmic design** and **new modules** to integrate them and achieve state-of-the-art results. As shown in Table 2 of the manuscript, the **straightforward combination** of HQ and LQ features **does not improve performance**. Thus, we introduce the **association stage** and the **feature fusion** modules to exploit the information from both domains effectively. The performance of the proposed model **significantly outperforms** state-of-the-art approaches, as shown in the following table.



| Ablation Study           | LMD &darr; | NIQE   &darr; |
| ------------------------ | ---------- | ------------- |
| Naively combine features | 4.258      | 4.297         |
| Association & Fusion     | **4.019**  | **3.815**     |



* We propose an **additional LQ reconstruction path** to specifically encode the LQ domain information, which **does not exist in prior works**. Furthermore, we provide a **new perspective** to handle the LQ domain information better. We show the **effectiveness of our additional LQ path** with extensive experiments. How to better deal with the LQ input, the only source of information in this task, to assist the restoration process is crucial. Our contributions in **providing new encoding strategies and viewpoints to handle LQ information** are essential to advance the field.

* We demonstrate the novel components of our work by comparing with RestoreFormer [1] and CodeFormer [2], whose network building blocks mostly come from VQGAN [3] and Transformer [4]. Although these models are mostly built upon prior works, they show great empirical results can be achieved by **novel and essential integration**. Such works are important and make significant contributions. We use these examples to emphasize that **sufficient novel components** are introduced in proposing an additional LQ encoder to specifically deal with the LQ information and demonstrate its **effectiveness** with extensive experiments. We hope reviewers can appreciate the contribution of this work from this perspective.

* We show the ability of our method in real-world datasets with **significant improvements** over SOTA methods in both quantitative and qualitative evaluation. For the qualitative results, please refer to our manuscript and supplementary material. For the quantitative results, please refer to the following tables.

| LFW-Test          | FID &darr; | NIQE   &darr; |
| ----------------- | ---------- | ------------- |
| RestoreFormer [1] | 48.412     | 4.168         |
| CodeFormer [2]    | 52.350     | 4.482         |
| VQFR [5]          | 50.712     | 3.589         |
| DAEFR (Ours)      | **47.532** | **3.552**     |

| WIDER-Test        | FID &darr; | NIQE   &darr; |
| ----------------- | ---------- | ------------- |
| RestoreFormer [1] | 49.839     | 3.894         |
| CodeFormer [2]    | 38.798     | 4.164         |
| VQFR [5]          | 44.158     | **3.054**     |
| DAEFR (Ours)      | **36.720** | 3.655         |

[1] RestoreFormer: High-Quality Blind Face Restoration from Undegraded Key-Value Pairs. CVPR 2022.

[2] Towards Robust Blind Face Restoration with Codebook Lookup Transformer. NeurIPS 2022.

[3] Taming Transformers for High-Resolution Image Synthesis. CVPR 2021.

[4] Attention Is All You Need. NeurIPS 2017.

[5] VQFR: Blind Face Restoration with Vector-Quantized Dictionary and Parallel Decoder. ECCV 2022.

Reviewer comment (Evaluation):
In Table 2, it seems like all the alter methods outperform the proposed method in terms of LPIPS. Please give discussions or visualizations to explain why this happens.

Author response:
**[Q2] In Table 2, it seems like all the altered methods outperform the proposed method in terms of LPIPS. Please give discussions or visualizations to explain why this happens.**


The LPIPS calculation involves using deep neural networks to analyze image features and then calculating a distance or similarity score based on these features. In our case, we input entire images, including background elements that are not our primary restoration focus. These background elements can affect the LPIPS score.

In this ablation study, our main goal is to demonstrate significant improvements in identity metrics. Additionally, we present qualitative results in Fig. 5 of the manuscript, showcasing the best visual outcomes achieved with our final setting.

Reviewer comment (Evaluation):
The paper does not delve into its limitations or potential factors for analysis, which would greatly enrich its discussion.

Author response:
We have provided visual results with large poses in Figure 20 of the supplementary material. While our method demonstrates robustness in most severe degradation scenarios, we also observe instances where it may fail, particularly in cases with large face poses. This can be expected as the FFHQ dataset contains few samples with large face poses, leading to a scarcity of relevant codebook features to effectively address such situations, resulting in less satisfactory restoration and reconstruction outcomes.

We include failure cases in the supplementary material to ensure a more comprehensive understanding of our method's limitations.

Reviewer comment (Evaluation):
The paper outperforms baseline methods in the downstream face recognition task. Could you provide a comprehensive explanation of these results?

Author response:
Our quantitative experiments demonstrate the strength of our method. The issue with identity evaluation on the synthetic CelebA-test dataset lies in the impracticality of achieving identical images in real-world scenarios. It only measures the likeness between the original image and the restored one. Real face recognition datasets comprise a vast number of images belonging to the same individual, presenting more significant challenges and closely mimicking real-world scenarios. These results confirm the effectiveness of our added LQ encoder, which extracts valuable details from the LQ domain, thereby significantly enhancing the restoration process.

Reviewer comment (Evaluation):
Omission of Limitation Discourse: The paper lacks a discussion regarding its limitations and possible factors for analysis.

Author response:
We have provided visual results with large poses in Figure 20 of the supplementary material. While our method demonstrates robustness in most severe degradation scenarios, we also observe instances where it may fail, particularly in cases with large face poses. This can be expected as the FFHQ dataset contains few samples with large face poses, leading to a scarcity of relevant codebook features to effectively address such situations, resulting in less satisfactory restoration and reconstruction outcomes.

We include failure cases in the supplementary material to ensure a more comprehensive understanding of our method's limitations.

Reviewer comment (Evaluation):
While the paper predominantly highlights the advantages of the proposed method, could you offer instances where the method encountered shortcomings or limitations?

Author response:
We have provided visual results with large poses in Figure 20 of the supplementary material. While our method demonstrates robustness in most severe degradation scenarios, we also observe instances where it may fail, particularly in cases with large face poses. This can be expected as the FFHQ dataset contains few samples with large face poses, leading to a scarcity of relevant codebook features to effectively address such situations, resulting in less satisfactory restoration and reconstruction outcomes.

We include failure cases in the supplementary material to ensure a more comprehensive understanding of our method's limitations.

Reviewer comment (Novelty):
Could you elaborate on the key distinction between DAEFR and CodeFormer?

Author response:
* Our method, DAEFR, introduces a crucial distinction by incorporating an additional encoder specifically designed for improved feature encoding in the LQ domain. This allows us to leverage essential information from the LQ images effectively. Moreover, DAEFR uses an association stage that effectively bridges the domain gap between the HQ and LQ domains.
* In contrast, CodeFormer adopts a different approach by utilizing a pretrained HQ encoder, which is then fine-tuned on LQ images. However, the domain gap still exists despite this adaptation, which may present certain challenges.
* By integrating the LQ encoder in DAEFR, we aim to encode LQ features better, thereby facilitating more accurate and refined results. This emphasis on enhancing the LQ encoding process demonstrates the critical distinction between DAEFR and CodeFormer, ultimately leading to improved performance and overcoming potential domain gap issues.

Reviewer comment (Experiments):
Can you provide further experimental details into the "Effectiveness of Low-Quality Feature from Auxiliary Branch" as examined in your ablation studies?

Author response:
To demonstrate the effectiveness of our auxiliary LQ branch, we conduct validated experiments of fusing LQ features with feature $Z^{c}\_{f}$ in the feature fusion and code prediction stage. These experiments involve extracting LQ features $Z^{c}\_{l}$ from the LQ codebook and adding a control module [1] [2]. Given a feature control module and feature scalar $s\_{lq}$, we can control the scale of the LQ feature $Z^{c}\_{l}$ to fuse with the feature $Z^{c}\_{f}$ before feeding to the HQ decoder. The network architecture is shown in Figure 10 in the supplementary material.

We conduct an additional training stage to enable the control module to fuse different features effectively. In our implementation, we fine-tune the control module for ten epochs and maintain a feature scalar value of $s\_{lq} =1$ during training.

[1] Recovering realistic texture in image super-resolution by deep spatial feature transform, CVPR 2018.

[2] Towards Robust Blind Face Restoration with Codebook Lookup Transformer. NeurIPS 2022.