Paper: Controlling Vision-Language Models for Multi-Task Image Restoration (ICLR-2024)

Reviewer comment (Experiments):
There are no justifications about what is embedded in the HQ content embedding. It is better to provide some comparisons with the HQ content embedding and the original image embedding.

Author response:
Thank you for this great suggestion. *In Figure 9(c) and 9(d) of the revised manuscript*, we have added the comparison of applying LQ content embeddings (the image embeddings obtained from the original CLIP) to the baseline model on both unified and degradation-specific image restoration tasks. Our DA-CLIP content embeddings clearly outperform the baseline of using original image embeddings. Moreover, the predicted HQ content embedding can also improve MSE-based approaches, please refer to the "General Response" (Table 2 and Table 3) for more details.

Reviewer comment (Experiments):
The effectiveness and necessity of the prompt learning module is not well discussed. It is better to provide some ablation studies to show the effectiveness of the prompt learning module compared with naive cross-attention.

Author response:
Thanks for this nice suggestion. Since the prompt is applied to all blocks of our model, we cannot substitute it with cross-attention which is really computationally inefficient (see *Section 3.2*). Instead, we included a comparison of replacing the prompt module with a simple network *in Figure 9(h) of the revised manuscript*, and the result demonstrates the added benefit of the prompt learning module.

Reviewer comment (Experiments):
The performance compared with `NAFNet+DA-CLIP` is not superior.

Author response:
We need to clarify that the `NAFNet+DA-CLIP` is also our method, which is proposed to prove that DA-CLIP can also improve the performance of vanilla MSE-based restoration approaches. For more details please refer to the "General Response" Table 2 and Table 3. Moreover, as we mentioned in the model evaluation section, this paper mainly focuses on perceptual metrics (which means realistic visual results) and our diffusion-based method performs the best in terms of LPIPS and FID. As further illustrated in the face inpainting case *in Figure 8*, our DA-CLIP generates correct details for the left eye region while other MSE-based methods can only produce blurry and smooth contents.

Reviewer comment (Evaluation):
Although the paper discussed the computation complexity in supplementary material, it only provides #params and FLOPS. It is known that the FLOPS is not a good metric for the computation complexity, especially for diffusion models which require multiple iteration steps. The paper should provide the inference time for the proposed method, and better in main paper.

Author response:
Thanks for pointing this out. It is true that diffusion-based methods require more inference time for multiple iterations. We have now added the runtime comparison in the below table. Note that MAXIM is implemented with the JAX GPU version, thus its comparison may not be fair. We then add the PromptIR and NAFNet for further reference. For the diffusion baseline IR-SDE, our DA-CLIP only adds 0.06s over 100 iterations (the prompt module and cross-attention also add 0.3s). For the direct MSE-based baseline NAFNet, there is no obvious increase in inference time.

**Table 4**. Comparison of the number of parameters and model computational efficiency. The flops and inference time are computed on the inpainting dataset.

|  Method  |  MIXIM  |  PromptIR  |  NAFNet  |  NAFNet + DA-CLIP  |  IR-SDE  |  IR-SDE + DA-CLIP  | 
|  ----  | ----  |  ----  | ----  |  ----  |  ----  | ----  | 
| \#Parameters | 14.1M | 33M | 67.8M | 86.1M + 125.2M | 36.2M | 48.9M + 125.2M |
| FLOPs | 216G | 158G | 63G | 65G + 0.5G | 117G | 118G + 0.5G |
| Inference time | 2.9s | 0.1s | 0.08s | 0.08s | 4.2s | 4.53s + 0.06s |

Reviewer comment (Experiments):
Experiments on more complex degradation scenarios, involving multiple concurrent degradation types, would emphasize the model's robustness and versatility.

Author response:
Since the mixed degradation dataset only contains a single degradation type label for each image, it is difficult for our method (and all other unified models) to restore multiple degradations of the same scene. Thank you for this comment, we would be very interested in creating models that can recover multiple concurrent degradations in future work. Moreover, as we mentioned in the "General Response" Table 1, DA-CLIP outperforms other unified methods on the light rain dataset (an out-of-distribution degradation), which also illustrates the robustness and versatility of our model.

Reviewer comment (Evaluation):
A deeper discussion about the text encoder's role in the performance could lead to a better understanding of the proposed framework.

Author response:
The text encoder is always fixed and only used for training DA-CLIP to predict degradation embeddings and HQ content embeddings. However, we can directly integrate the text encoder's embedding into the downstream model as a text-guided restoration process. The experiments are shown *in Figure 9(f) and Figure 9(g) in the revised manuscript*, in which we find that using ground truth text embeddings from the text encoder leads to a similar performance to that of using image embeddings from our DA-CLIP. However, text embedding requires access to the ground truth degradation label and clean caption for each image. Since we only have LQ images in testing, we choose to use image embeddings for all experiments.

Reviewer comment (Experiments):
The necessity of using VLM like CLIP in this paper is unclear. The proposed method does not seem to have much relationship with VLM like CLIP. The DA-CLIP serves as a degradation type classifier and a content classifier, and it seems possible to replace CLIP with simple CNNs such as ResNet. Given that the degradation types are quite limited, a text-encoder like CLIP could be replaced with simple classification. As for the captions, they could be replaced with clean image embeddings to perform contrastive learning.

Author response:
The main motivation for leveraging CLIP in this work is to provide meaningful semantic instructions to improve the restoration process. Previous works like AirNet [1] have tried to use a CNN encoder with contrastive learning as the degradation classifier, but its performance is much worse than our DA-CLIP, as illustrated *in Table 3 of our paper*. Particularly, CLIP is pretrained on billions of web-scale text-image pairs which can provide more accurate visual embeddings and stronger generalization ability than vanilla-trained CNNs or vision Transformers. As we mentioned in the "General Response" Table 1, our DA-CLIP performs much better than all other unified models on the unseen light rain dataset.

Reviewer comment (Novelty):
The use of a diffusion model for the restoration process is not well justified. Given the slow inference speed of diffusion models, it is unclear why they are used here, especially when the diffusion network is trained from scratch. Compared with `NAFNet+DA-CLIP`, the diffusion-based backbone does not have significant advantages.

Author response:
As we clarified in Q3, we focus on the diffusion-based model due to its strong ability to generate realistic textures and details (which means better visual results, as for the face inpainting example *in Figure 8*). Moreover, we clarify that our DA-CLIP also works well for vanilla MSE-based approaches, such as the `NAFNet+DA-CLIP`. Please refer to the "General Response" Table 2 and Table 3 for more details.

Reviewer comment (Theory):
It is questionable that the caption embedding can provide a high-quality image representation supervision for the content embedding. $e_c^T$ can indeed provide a semantic supervision, but there is no guarantee that it is a clean image representation. Therefore, I think the claim that the image encoder with the controller outputs high-quality content features is not rigorous. It seems that $e_c^I$ mainly serves to provide semantic instruction for the restoration network, especially for diffusion-based models.

Author response:
A key motivation for leveraging the pretrained VLM for image restoration is that VLMs like CLIP can accurately align image representations with corresponding caption embeddings. Note that the caption generated from an LQ image would be different from that of an HQ image. Such as the snow example shown *in Figure 4*, the caption from HQ image precisely describes the main content of the scene, which supervises our model to learn a snow-free HQ image representation from the LQ image. In addition, since we fixed the text encoder in training, matching the heavy snow scene to a snow-free caption embedding guarantees that the learned content embedding contains no snow information, which is called "clean" and "high-quality" in the context of image desnowing. 

We have further illustrated this by applying two different image content embeddings (HQ embedding from DA-CLIP and LQ embedding from the original CLIP) to the baseline *in Figures 9(c) and 9(d) of the revised submission*. The results show that our DA-CLIP content embedding can provide more useful supervision to the learning process on both unified and degradation-specific restoration.

Moreover, the learned content embedding $e_c^I$ can also benefit MSE-based restoration approaches. As we illustrated in the "General Response" Table 2 and Table 3, both NAFNet and PromptIR can be further improved by adding our DA-CLIP content embeddings.

Reviewer comment (Experiments):
The used experimental setup is too simple to demonstrate the superiority of this complex method. It is not difficult for a unified network, e.g., a vanilla version Restormer, to deal with the all-in-one image restoration setting with a specific degradation level for each degradation type.

Author response:
Although we can directly train a vanilla network across multiple degradations, its performance is inferior to approaches that learn discriminative degradation context explicitly [1,2]. As illustrated in the "General Response" Table 2, combining the vanilla model NAFNet with our DA-CLIP significantly outperforms the original NAFNet on the unified restoration task. In addition, as nicely suggested, we have added the Restormer experiment *in Table 3 of the revised manuscript* (also in the "General Response" Table 2). It is worth noting that PromptIR is built upon Restormer with additional degradation prompt modules, which largely improves the unified restoration results.

---

*[1] All-in-one Image Restoration for Unknown Degradations Using Adaptive Discriminative Filters for Specific Degradations. CVPR 2023.*

*[2] PromptIR: Prompting for All-in-One Blind Image Restoration. NeurIPS 2023.*

Reviewer comment (Experiments):
This paper do not provide the experiment about the generalization ability of the proposed method. I do some tests using the code provided by the authors, and the results show that the model cannot deal with out-of-distribution degradations as well as OOD degradation levels well. It is not surprising as the used degradation model is too simple. This also reflects that $e_c^I$ is not always a high-quality image representation.

Author response:
Due to the data source limitation, almost all unified models struggle to deal with OOD degradations. In the "General Response" Table 1, we provided an experiment of testing all models on the Rain100L dataset which contains 100 light rain images. Although our method is only trained on heavy rain images (Rain100H), it still performs well and significantly surpasses all other unified approaches. Moreover, we have added visual results on real-world images *in our revised submission (Figure 16)*, which also indicate that our method has a better generalization ability compared to other unified methods. In our future work, we could potentially add some data augmentation steps (e.g., resize, random noise, and blur) in training to make our model more practical, similar to other blind super-resolution approaches such as Real-ESRGAN. We would be also very interested in creating models that perform well on OOD degradation types and levels.

Reviewer comment (Experiments):
This method may be difficult to handle tasks with different degradation levels, because it is difficult to describe the specific degradation level in texts. Since the authors do not provide relevant experiments, the potential of this method to handle multiple degradation levels is still questionable. As far as the current results are concerned, the approach is not practical enough.

Author response:
It is true that DA-CLIP cannot describe the degradation levels in texts. However, benefiting from the predicted HQ content embedding, our method is still able to handle different degradation levels. For example, both the hazy and snowy datasets contain mixed images across multiple degradation levels. Moreover, in the "General Response", Table 1 also illustrates that our DA-CLIP generalizes better to an unseen degradation level compared to other unified models.

Reviewer comment (Theory):
Do both $e_c^I$ and $e_d^I$ have an important impact on the restoration performance? Intuitively, it is reasonable for diffusion-based models as $e_c^I$ controls the content and $e_d^I$ indicates the degradation. However, it seems not reasonable for mse-based models to use $e_c^I$ in the restoration process.

Author response:
In our experiments, both $e_c^I$ and $e_d^I$ are important and can improve the performance of unified models as illustrated *in Figures 9(a), 9(c) and 9(d) in the revised manuscript*. Moreover, the two embeddings can be applied also to MSE-based restoration models, as illustrated by the ablation experiments in Table 2 and Table 3 in the "General Response". The results show that adding DA-CLIP content embedding $e_c^I$ to NAFNet and PromprIR improves their performance, illustrating the importance of DA-CLIP content embedding on MSE-based approaches.

Reviewer comment (Experiments):
What performance can be achieved by directly providing the semantic text prompt and the degradation type prompt to train the all-in-one image restoration diffusion model?

Author response:
The model with ground truth degradation text embeddings has a similar performance to the predicted degradation embeddings, as illustrated *in Figure 9(e) of the revised manuscript*. Moreover, we further added a comparison of providing both ground truth (GT) degradation type and semantic caption text embeddings to the baseline diffusion model *in Figure 9(f) and 9(g)*. The model with both GT degradation and caption text embeddings outperforms the baseline that only has degradation text embeddings (*Figure 9(f)*). However, the method with GT text embeddings performs slightly worse than our model which uses predicted image embeddings (*Figure 9(g)*).

Reviewer comment (Experiments):
What performance can be achieved by directly training a vanilla Restormer under the same all-in-one setting?

Author response:
As shown in the "General Response" Table 2, Restormer performs slightly better than NAFNet but is inferior to PromptIR. In fact, PromptIR is built on top of Restormer, simply adding degradation prompt modules. In addition, Table 2 also shows that adding DA-CLIP content embedding can further improve the performance across all metrics.

Reviewer comment (Experiments):
In Figure 1, DA-CLIP achieves surprisingly high accuracy in ten degradation types. How are these experiments set up? In contrast, CLIP performs poorly in many types. What prompts do the authors use for classifying degradations in CLIP?

Author response:
In our experiments, we use the ten degradation types as label texts, and the prompt is set to "a *[degradation type]* photo". Since we freeze the text encoder in training, the same text embedding would be used for both DA-CLIP and the original CLIP to match low-quality images. We provide additional experiments of directly retraining or finetuning CLIP on our dataset *in Table 4 of the paper*. Our DA-CLIP achieves a similar accuracy with CLIP + finetune, while it can also output clean content embedding to further improve downstream image restoration performance.

Reviewer comment (Evaluation):
In Figure 6, PromptIR is comparable or even better than the proposed DA-CLIP in most tasks on fidelity metrics.

Author response:
Note that PromptIR is a purely MSE-based approach that achieves high fidelity metric scores (PSNR and SSIM) but will produce perceptually unsatisfying results[1]. Our method, similar to other diffusion approaches[2], mainly focuses on generating realistic images and achieves the best perceptual performance (LPIPS and FID) *in Figure 6 and Table 3 of the paper*. As can be seen in the inpainting example in *Figure 8*, the face generated by PromptIR loses all details on the left eye region, while our method predicts more details and produces relatively good visual results. In addition, simply adding the HQ content embedding of DA-CLIP can further improve PromptIR across all metrics (please refer to the "General Response" Table 2 for more details).

Reviewer comment (Evaluation):
In Table 2(c), the PSNR of DA-CLIP highly deviates from that of MAXIM. In addition, the results on task-specific restoration do not show a clear benefit of using a universal model for all tasks. It is believed that the merit of universal models is that different tasks can benefit each other, or at least be helpful in generalization to new domains.

Author response:
Similar to PromptIR, MAXIM is also an MSE-based method and is designed for degradation-specific image restoration. Although it surpasses DA-CLIP by 2dB in terms of PSNR on the GoPro dataset, our method still obtains the best perceptual scores (0.058/6.15 in LPIPS/FID compared to 0.089/11.57) and produce visually better results. In the "General Response" Table 3, we have further added an ablation experiment of applying DA-CLIP to NAFNet on two single-degradation tasks: image deraining and low-light enhancement. The results show that DA-CLIP clearly improves the performance of NAFNet, illustrating the effectiveness of our method on degradation-specific restoration.