Paper: Harnessing Joint Rain-/Detail-aware Representations to Eliminate Intricate Rains (ICLR-2024)

Reviewer comment (Theory):
Figure 1(b) shows one observation of this paper that the image is gradually dominated by rain when rain density increases, which, however, sounds superficial. The resulting (from this observation) learning of a joint embedding space for both rain and detail information is based on contrastive learning, while it is not explained why using contrastive learning here as it seems that rain density is the most important factor. Meanwhile, the paper simply says it is distinct from previous instance-discriminative contrastive learning (He et al., 2020) without explanation.

Author response:
We want to clarify that the observations in Figure 1(b) are non-trivial, which may sound superficial to some extent. In fact, **we proposed a measure to quantitatively illustrate the relationships between rain/detail cues and rain density.** Second, *although the rain factor becomes dominant when rain density increases, the number of heavy rainy images drops dramatically, as illustrated in Figure 1 (a).* Please note that rainy images with relatively small density account for a large proportion. Therefore, **to learn a model for diverse rainy images, both rain-aware and detail-aware information should be considered.** In contrast, most recent contrastive learning-based [1,2,3] focus on exploring degradation-aware information, where detail-aware information is overlooked. Therefore, our findings from Figure 1 (a) & (b) provide insight into considering joint rain-/detail-aware representations for image deraining. In practice, our ablation study on negative exemplar types also substantiates that the detail-aware information can enhance deraining performance on Rain800 (please refer to Table 5 for details). Additionally, **only considering rain-aware representations may cause a trivial solution for the embedding space, where only heavy rains are distinguishable** (please refer to Figure 6 (c) & (d) for details).

Since almost all rain datasets only provide ground truth as labels, we can only learn the representations in an unsupervised manner. Therefore, considering the success of contrastive learning-based methods, we designed a novel contrastive learning strategy to learn joint rain-/detail-aware representations. **Note that the contrastive learning method in (He et al., 2020) [4] is only content-related discriminative, neglecting the rain factor; thus, it cannot be utilized to extract rain-aware representations.**

[1]. Xiang Chen, et al. "Unpaired deep image deraining using dual contrastive learning." in CVPR 2022.

[2]. Longguang Wang, et al. "Unsupervised degradation representation learning for blind super-resolution." in CVPR 2021.

[3]. Boyun Li, et al. "All-in-one image restoration for unknown corruption." in CVPR 2022.

[4]. Kaiming He, et al. "Momentum contrast for unsupervised visual representation learning." in CVPR 2020.

Reviewer comment (Writing):
The Introduction mentions that the embedding space of this paper is not instance-level discriminative, which is hard to understand as the paper aims to learn instance-specific deraining features (If not discriminative enough, how can one model learn the specific information?). The second point it mentions is that the background content is not discriminative, which I do not know whether it is correct, and it does not explain why it is not suitable for a standard contrastive learning strategy. After I read the method section, it seems to me that the main difference is to construct negative examples based on rain types (selected from another dataset) and Gaussian blurred background. The authors may correct me if I misunderstood, but it seems to me that the discussion in the introduction does not match what they do in the method.

Author response:
Thank you for pointing out this ambiguity, and we apologize for not clearly clarifying our motivation for employing contrastive learning, as well as the differences from other methods. We want to address the ambiguity caused by our original explanation (**We have revised this part in the *Introduction*, where we provide an in-depth discussion on existing contrastive learning methods, and we also remove all instance-specific phrases for better understanding**). The embedding space is not instance-level discriminative, meaning that **rainy images are not discriminative based on image content due to the coupled rain.** Therefore, the content-based contrastive learning approach [1] is rendered inapplicable. In fact, **our goal is to extract both rain- and detail-aware information that characterizes the rain component and background component. This rain-/detail-aware information, in turn, provides guidance to adaptively separate the rain from the rainy input.** Therefore, we need to learn an encoder that can well perceive different rains and background details in an unsupervised manner. Inspired by the success of unsupervised contrastive learning-based image restoration methods, we have designed a novel contrastive learning strategy that enables us to extract both rain- and detail-related representations.

Existing contrastive learning methods mainly focus on extracting only the degradation factor [2,3,4] or aim at discriminating between rainy and clean images [5]. Therefore, these methods cannot effectively extract joint rain-/detail-related information from the rainy input. Moreover, the rain-aware negatives are not simply selected from another dataset, since *images from different datasets may share similar rain patterns*. Instead, **we maintain a *rain layer bank* during training, which is used to retrieve the most dissimilar rain pattern to construct negative exemplars.** Based on the proposed contrastive learning strategy, we can quantitatively assess the importance of rain and detail cues for deraining.

To clearly underline the novelty of the proposed contrastive learning, we have revised the section on *Introduction* to better explain the limitations of existing approaches, as well as to clarify the differences from these approaches.

Reviewer comment (Theory):
The paper uses Gaussian blur to construct negative exemplars of the background image, which needs further justification as it may not represent the degraded backgrounds. Usually when rain streaks are large or dense, there are more occlusions than blur, which the Gaussian blur may not model.

Author response:
Please note that Gaussian blur is used to **blur the details in the ground truth**, which serves as negative exemplars and should be pushed apart from the rainy image. When the rain streaks are large or dense, more occlusions than blur occur, which decreases the discriminability of details, as the reviewer pointed out. **But recall that when rain streaks are large or dense, the rain-aware information dominates; hence, the effect caused by the limitation of Gaussian blur can be neglected.** **With the help of the rain-aware negatives and detail-aware negatives, the proposed contrastive learning strategy can function well with either slight rain or dense rain.** In practice, we find that Gaussian blur works well.

[1]. Kaiming He, et al. "Momentum contrast for unsupervised visual representation learning." in CVPR 2020.

[2]. Xiang Chen, et al. "Unpaired deep image deraining using dual contrastive learning." In CVPR 2022.

[3]. Longguang Wang, et al. "Unsupervised degradation representation learning for blind super-resolution." in CVPR 2021.

[4]. Boyun Li, et al. "All-in-one image restoration for unknown corruption." in CVPR 2022.

[5]. Yuntong Ye, et al. "Unsupervised deraining: Where contrastive learning meets self-similarity." in CVPR 2022.

Reviewer comment (Novelty):
The related work section should be expanded. There are certainly several rain removal methods not cited, especially for heavy rain restoration, although they do not combine multiple datasets for training. The modulation is one of the key techniques in the proposed method but there is no literature review on it.

Author response:
Thanks for your constructive advice. We have added four methods: a heavy rain removal method [1], two semi-supervised methods Syn2Real [2] and MOSS [3], and one unsupervised method NLCL [4]. Additionally, we have included a literature review of modulation techniques in image restoration entitled "Image Restoration with Modulation" in the *Related Work section.*

Reviewer comment (Novelty):
In Table 1, the performance gain over DRSformer (Chen et al., 2023) is rather limited. The paper needs to show more advantages, otherwise, it seems not worth doing considering the efforts and the performance gains.

Author response:
We acknowledge that the performance gain over DRSformer is marginal, mainly due to the Mixture of Experts (MoEs) in it. **However, we may have underestimated the potential of the proposed CoIC.** Considering suggestions from other reviewers, we have added a real-world dataset SPAData [5] to tune DRSformer for about 105k iterations. We tabulate the quantitative results below (please see details in Table 2):

| Methods            | Rain200L  | Rain200H  | Rain800   | DID-Data  | DDN-Data  | SPAData   |
| ------------------ | --------- | --------- | --------- | --------- | --------- | --------- |
| DRSformer w/o CoIC | 39.32     | 29.27     | 28.85     | 34.91     | 33.71     | 45.46     |
| DRSformer w/ CoIC  | **39.70** | **30.31** | **29.73** | **35.02** | **33.94** | **46.03** |

Note that **the proposed method has achieved a 1.04 dB, 0.88 dB, and 0.57 dB improvement on Rain200H, Rain800, and SPAData.** Obviously, the MoEs cannot handle the mixed synthetic and real-world rains effectively. In contrast, the proposed CoIC enables us to obtain a comprehensive deraining model adept at deraining diverse synthetic and real-world rains. Furthermore, we have also observed a significant deraining ability boost on RealInt. Please see our anonymous GitHub page: https://anonymous.4open.science/r/CoIC-730F/

With the success after tuning DRSformer with CoIC on SPAData, we provide high-quality real-world deraining results in Figure 4 and Figure 14. For convenience, please review it on our anonymous GitHub page: https://anonymous.4open.science/r/CoIC-730F/

In summary, the results of tuning on SPAData can well substantiate the effectiveness of the proposed CoIC. **In the future, we plan to extend the proposed method for image deraining under diverse rains coupled with fog, dim light, blur, and noise.** 

[1]. Ruoteng Li et al. "Heavy rain image restoration: Integrating physics model and conditional adversarial learning." in CVPR 2019.

[2]. Yasarla, Rajeev, et al. "Syn2real transfer learning for image deraining using gaussian processes." in CVPR 2020.

[3]. Huaibo Huang, et al. "Memory oriented transfer learning for semi-supervised image deraining." in CVPR 2021.

[4]. Yuntong Ye, et al. "Unsupervised deraining: Where contrastive learning meets self-similarity." in CVPR 2022.

[5]. Tianyu Wang et al. "Spatial attentive single-image deraining with a high quality real rain dataset." in CVPR 2019.

Reviewer comment (Evaluation):
The visual results are not impressive, as we may easily see the remained rain streaks in, e.g., Figure 4 and Figure 12. Although there are improvements by comparing the results of methods with and without CoIC, these visual results are still failure cases (e.g., due to obvious rain streaks, in Figure 12, (g) is comparable to or even better than (h)). Combined with Table 1, this really makes others wonder about the necessity of using the proposed approach.

Author response:
We acknowledge that the performance gain over DRSformer is marginal, mainly due to the Mixture of Experts (MoEs) in it. **However, we may have underestimated the potential of the proposed CoIC.** Considering suggestions from other reviewers, we have added a real-world dataset SPAData [5] to tune DRSformer for about 105k iterations. We tabulate the quantitative results below (please see details in Table 2):

| Methods            | Rain200L  | Rain200H  | Rain800   | DID-Data  | DDN-Data  | SPAData   |
| ------------------ | --------- | --------- | --------- | --------- | --------- | --------- |
| DRSformer w/o CoIC | 39.32     | 29.27     | 28.85     | 34.91     | 33.71     | 45.46     |
| DRSformer w/ CoIC  | **39.70** | **30.31** | **29.73** | **35.02** | **33.94** | **46.03** |

Note that **the proposed method has achieved a 1.04 dB, 0.88 dB, and 0.57 dB improvement on Rain200H, Rain800, and SPAData.** Obviously, the MoEs cannot handle the mixed synthetic and real-world rains effectively. In contrast, the proposed CoIC enables us to obtain a comprehensive deraining model adept at deraining diverse synthetic and real-world rains. Furthermore, we have also observed a significant deraining ability boost on RealInt. Please see our anonymous GitHub page: https://anonymous.4open.science/r/CoIC-730F/

With the success after tuning DRSformer with CoIC on SPAData, we provide high-quality real-world deraining results in Figure 4 and Figure 14. For convenience, please review it on our anonymous GitHub page: https://anonymous.4open.science/r/CoIC-730F/

In summary, the results of tuning on SPAData can well substantiate the effectiveness of the proposed CoIC. **In the future, we plan to extend the proposed method for image deraining under diverse rains coupled with fog, dim light, blur, and noise.** 

[1]. Ruoteng Li et al. "Heavy rain image restoration: Integrating physics model and conditional adversarial learning." in CVPR 2019.

[2]. Yasarla, Rajeev, et al. "Syn2real transfer learning for image deraining using gaussian processes." in CVPR 2020.

[3]. Huaibo Huang, et al. "Memory oriented transfer learning for semi-supervised image deraining." in CVPR 2021.

[4]. Yuntong Ye, et al. "Unsupervised deraining: Where contrastive learning meets self-similarity." in CVPR 2022.

[5]. Tianyu Wang et al. "Spatial attentive single-image deraining with a high quality real rain dataset." in CVPR 2019.

Reviewer comment (Novelty):
The main concern is the necessity of the proposed method. In fact, when the deraining model is strong enough, it may be able to learn the differences between datasets. For example, in Table 1, the proposed method has limited performance improvement on DRSformer and DGUNet.

Author response:
The performance improvement in Table 1 for DGUNet and DRSformer seems marginal, primarily because these models may be capable enough to process synthetic datasets. For instance, DRSformer contains a Mixture of Experts (MoEs) that enables it to tolerate different rains. **However, the potential of the proposed CoIC may be underestimated only based on the results from synthetic datasets.** As suggested by other reviewers, we have added a real-world dataset SPAData [1] to fine-tune the pre-trained DRSformer for another 105k iterations. Below are the quantitative results in terms of PSNR (Details are provided in Table 2):

| Methods            | Rain200L  | Rain200H  | Rain800   | DID-Data  | DDN-Data  | SPAData   |
| ------------------ | --------- | --------- | --------- | --------- | --------- | --------- |
| DRSformer w/o CoIC | 39.32     | 29.27     | 28.85     | 34.91     | 33.71     | 45.46     |
| DRSformer w/ CoIC  | **39.70** | **30.31** | **29.73** | **35.02** | **33.94** | **46.03** |

The results indicate that even with MoEs, DRSformer directly tuned on SPAData suffers heavy performance drops on Rain200L, Rain200H, and Rain800 datasets. However, **with the help of CoIC, DRSformer could achieve improvements of 1.04dB, 0.88dB, and 0.57dB on Rain200H, Rain800, and the real-world SPAData datasets.** This observation reveals that we could train a comprehensive model targeted at deraining both synthetic and real-world data using CoIC. Additionally, we also obtained a remarkable improvement in deraining ability on RealInt, displayed in Figure 4 and Figure 14. For your convenience, please review the results on our anonymous GitHub pages: https://anonymous.4open.science/r/CoIC-730F/

Reviewer comment (Evaluation):
It would be better to show the increase in the number of parameters and changes in inference time brought by the proposed method.

Author response:
We summarize the number of parameters, FLOPs, and inference time for BRN, RCDNet, DGUNet, IDT, and DRSformer in the table below (Also included in *Appendix A.3*):

| Model name | input size | #p w/o CoIC (M) | #$\Delta$p  (M) | FLOPs w/o CoIC (G) | FLOPs w/CoIC (G) | Time w/o CoIC (ms) | Time w/ CoIC (ms) |
| ---------- | ---------- | :-------------: | --------------- | ------------------ | ---------------- | ------------------ | ----------------- |
| BRN        | 512x512    |      0.38       | 0.21            | 392.9              | 393.3            | 332.2$\pm$38.1     | 364.2$\pm$33.0    |
| RCDNet     | 512x512    |      2.98       | 2.11            | 389.0              | 389.5            | 351.6$\pm$0.2      | 384.8$\pm$2.2     |
| DGUNet     | 512x512    |      3.63       | 1.62            | 396.8              | 397.2            | 161.0$\pm$5.5      | 198.8$\pm$6.8     |
| IDT        | 128x128    |      16.42      | 2.53            | 7.3                | 7.6              | 59.8$\pm$2.7       | 83.2$\pm$10.5     |
| DRSformer  | 512x512    |      33.67      | 14.12           | 440.8              | 441.1            | 833.6$\pm$0.5      | 886.7$\pm$11.0    |

Generally, **the increased parameters are related to the intrinsic architecture of models** since the proposed CoIC performs layer-wise modulation. **It is noteworthy that the performance improvement is not mainly from the increased parameters**. Note that IDT equipped with CoIC with fewer increased parameters can result in much better performance improvement when compared to DRSformer in Table 1, where more parameters are brought by CoIC for DRSformer. Moreover, the changes in FLOPs and inference time (about 30ms) are almost the same for all models, which means that **the additional changes are mainly from the encoder but not the modulation process.**

Reviewer comment (Reproducibility):
It is not clear how to select the positive and negative exemplars. The author should discuss the differences with relevant contrastive learning-based deraining methods [1-2].  
[1] Chen et al, Unpaired deep image deraining using dual contrastive learning, CVPR2022.  
[2] Ye et al, Unsupervised deraining: Where contrastive learning meets self-similarity, CVPR2022.

Author response:
We apologize for not clearly clarifying the differences between the proposed contrastive learning and existing contrastive learning methods. Our findings in Figure 1 suggest that learning joint rain-/detail-aware representations is meaningful for training models on mixed multiple datasets. This implies the need to train an encoder capable of discriminating different rains as well as background details. Hence, **the negative exemplars should contain exemplars with different rains and exemplars with different background details**. However, the recent method DCD-GAN [1] only considers the differences between rain and a clean background, which cannot perceive different rain types as well as background details. Additionally, the unsupervised method NLCL [2] also considers the differences between rain and a clean background. Although [2] introduces location contrastive learning to constrain the restored background to be the same as the ground truth, it cannot extract detail-aware information from the rainy input. Hence, [2] cannot be used to provide joint rain-/detail-aware guidance for adaptive image deraining. To clearly elaborate on the differences between our CoIC and recent contrastive learning methods, **we have added an in-depth discussion in the section of *Introduction***.

We also trained the unsupervised method DCD-GAN on mixed synthetic datasets for more than 1M iterations. However, it failed to process intricate rains from multiple datasets. Below, we present its PSNR metrics (Details can be found in Table 1):

| Methods          | Rain200L  | Rain200H  | Rain800   | DID-Data  | DDN-Data  |
| ---------------- | --------- | --------- | --------- | --------- | --------- |
| DCD-GAN          | 21.64     | 16.04     | 19.52     | 21.28     | 21.60     |
| DRSformer + CoIC | **39.81** | **30.50** | **29.92** | **35.01** | **33.94** |

In comparison to the proposed CoIC, DCD-GAN struggles on all synthetic benchmark datasets.

Reviewer comment (Experiments):
I suggest the author conduct further validation on the mixed dataset Rain13K.

Author response:
Thank you for this constructive suggestion. Since marginal improvement of CoIC on DRSformer has been observed in Table 1, it may be helpful to train it on another mixed dataset to validate its efficacy. However, as pointed out by reviewer nGEY, the Rain13K dataset could be regarded as one entity. Therefore, to further explore the potential of CoIC, **we have added a real-world dataset SPAData [3] to fine-tune DRSformer**. Surprisingly, **we observe that the tuned model using CoIC can achieve much better performance on both synthetic and real-world SPAData datasets**. We present the quantitative results in terms of PSNR below (Details are provided in Table 2):

| Methods            | Rain200L  | Rain200H  | Rain800   | DID-Data  | DDN-Data  | SPAData   |
| ------------------ | --------- | --------- | --------- | --------- | --------- | --------- |
| DRSformer w/o CoIC | 39.32     | 29.27     | 28.85     | 34.91     | 33.71     | 45.46     |
| DRSformer w/ CoIC  | **39.70** | **30.31** | **29.73** | **35.02** | **33.94** | **46.03** |

Moreover, **the fine-tuned DRSformer with CoIC can efficiently remove complex real rain streaks in RealInt**. We have showcased high-quality real-rain removal results in Figure 4 and Figure 14. For your convenience, please review them on our anonymous GitHub page: https://anonymous.4open.science/r/CoIC-730F/.

Reviewer comment (Presentation):
The figures included in the manuscript are rendered with small font sizes and low resolution, which makes them challenging for reviewers to examine and comprehend. I recommend revising the figures to meet these requirements to improve the overall quality and accessibility of the visuals in the manuscript.

Author response:
We apologize for any inconvenience caused by the small font size in the figures, which may have posed readability issues. In the revised manuscript, we have adjusted the figures and increased the font size to enhance the reading experience.

Reviewer comment (Experiments):
The comparisons are inadequate. There is rich literature in semi-supervised and continual learning that focuses on representation learning for different types of rain and image representations. Authors failed to compare with the existing methods.

Author response:
We first added the semi-supervised method Syn2Real [1], MOSS [2], and the unsupervised method DCD-GAN [3] to our *Introduction* and literature review in *Related Work*. Since the official training code for MOSS [2] is unavailable, we compare the proposed CoIC with Syn2Real and DCD-GAN. Both Syn2Real and DCD-GAN are trained for more than 1M iterations (much longer than BRN, RCDNet, DGUNet, IDT, and DRSformer). However, we observed that both Syn2Real and DCD-GAN struggle when faced with intricate rainy images from multiple datasets, yielding unsatisfactory results on Rain200L, Rain200H, Rain800, DID-Data, and DDN-Data datasets. We tabulate the PSNR metrics below for direct comparison (Details are in Table 1 of the revised manuscript):

| Methods          | Rain200L  | Rain200H  | Rain800   | DID-Data  | DDN-Data  |
| ---------------- | --------- | --------- | --------- | --------- | --------- |
| Syn2Real         | 30.83     | 17.21     | 24.85     | 26.71     | 29.15     |
| DCD-GAN          | 21.64     | 16.04     | 19.52     | 21.28     | 21.60     |
| DRSformer + CoIC | **39.81** | **30.50** | **29.92** | **35.01** | **33.94** |

From the above results, we observe that learning a comprehensive deraining model on mixed datasets is challenging for semi-supervised and unsupervised methods. The proposed CoIC can efficiently assist models in learning much better on multiple datasets.

Additionally, we have provided a demo on our anonymous GitHub page: https://anonymous.4open.science/r/CoIC-730F/

[1]. Yasarla, Rajeev, et al. "Syn2real transfer learning for image deraining using gaussian processes." in CVPR 2020.

[2]. Huaibo Huang, et al. "Memory oriented transfer learning for semi-supervised image deraining." in CVPR 2021.

[3]. Xiang Chen, et al. "Unpaired deep image deraining using dual contrastive learning." In CVPR 2022.

Reviewer comment (Experiments):
Memory Oriented Transfer Learning for Semi-Supervised Image Deraining, CVPR 2021, proposes a representation learning based approach where they learn basis vectors (which they call memory) to represent the rain and adapt them for different datasets to minimize the differences between datasets. The proposed CoI-C is also doing something similar to this, so it will be easy for the reader to understand the benefits of CoI-C if authors can compare with this method.

Author response:
We first added the semi-supervised method Syn2Real [1], MOSS [2], and the unsupervised method DCD-GAN [3] to our *Introduction* and literature review in *Related Work*. Since the official training code for MOSS [2] is unavailable, we compare the proposed CoIC with Syn2Real and DCD-GAN. Both Syn2Real and DCD-GAN are trained for more than 1M iterations (much longer than BRN, RCDNet, DGUNet, IDT, and DRSformer). However, we observed that both Syn2Real and DCD-GAN struggle when faced with intricate rainy images from multiple datasets, yielding unsatisfactory results on Rain200L, Rain200H, Rain800, DID-Data, and DDN-Data datasets. We tabulate the PSNR metrics below for direct comparison (Details are in Table 1 of the revised manuscript):

| Methods          | Rain200L  | Rain200H  | Rain800   | DID-Data  | DDN-Data  |
| ---------------- | --------- | --------- | --------- | --------- | --------- |
| Syn2Real         | 30.83     | 17.21     | 24.85     | 26.71     | 29.15     |
| DCD-GAN          | 21.64     | 16.04     | 19.52     | 21.28     | 21.60     |
| DRSformer + CoIC | **39.81** | **30.50** | **29.92** | **35.01** | **33.94** |

From the above results, we observe that learning a comprehensive deraining model on mixed datasets is challenging for semi-supervised and unsupervised methods. The proposed CoIC can efficiently assist models in learning much better on multiple datasets.

Additionally, we have provided a demo on our anonymous GitHub page: https://anonymous.4open.science/r/CoIC-730F/

[1]. Yasarla, Rajeev, et al. "Syn2real transfer learning for image deraining using gaussian processes." in CVPR 2020.

[2]. Huaibo Huang, et al. "Memory oriented transfer learning for semi-supervised image deraining." in CVPR 2021.

[3]. Xiang Chen, et al. "Unpaired deep image deraining using dual contrastive learning." In CVPR 2022.

Reviewer comment (Experiments):
Syn2Real Transfer Learning for Image Deraining using Gaussian Processes, CVPR 2020, proposes a Gaussian process based pseudo labeling approach that enables the encoder to learn representation of rain and image using generated pseudo labels. In Syn2Real they formulate the joint Gaussian distribution to generate a pseudo label for the unknown or unlabeled image, by selecting the k-nearest labeled images and maximizing correlation between unlabeled and labeled similar type of rain images, and also minimizing correlation between k-nearest farthest labeled images and unlabeled and labeled similar type of rain images. This approach can be compared to the proposed contrastive loss and CoI-C approach, to understand the benefits of CoI-C.

Author response:
We first added the semi-supervised method Syn2Real [1], MOSS [2], and the unsupervised method DCD-GAN [3] to our *Introduction* and literature review in *Related Work*. Since the official training code for MOSS [2] is unavailable, we compare the proposed CoIC with Syn2Real and DCD-GAN. Both Syn2Real and DCD-GAN are trained for more than 1M iterations (much longer than BRN, RCDNet, DGUNet, IDT, and DRSformer). However, we observed that both Syn2Real and DCD-GAN struggle when faced with intricate rainy images from multiple datasets, yielding unsatisfactory results on Rain200L, Rain200H, Rain800, DID-Data, and DDN-Data datasets. We tabulate the PSNR metrics below for direct comparison (Details are in Table 1 of the revised manuscript):

| Methods          | Rain200L  | Rain200H  | Rain800   | DID-Data  | DDN-Data  |
| ---------------- | --------- | --------- | --------- | --------- | --------- |
| Syn2Real         | 30.83     | 17.21     | 24.85     | 26.71     | 29.15     |
| DCD-GAN          | 21.64     | 16.04     | 19.52     | 21.28     | 21.60     |
| DRSformer + CoIC | **39.81** | **30.50** | **29.92** | **35.01** | **33.94** |

From the above results, we observe that learning a comprehensive deraining model on mixed datasets is challenging for semi-supervised and unsupervised methods. The proposed CoIC can efficiently assist models in learning much better on multiple datasets.

Additionally, we have provided a demo on our anonymous GitHub page: https://anonymous.4open.science/r/CoIC-730F/

[1]. Yasarla, Rajeev, et al. "Syn2real transfer learning for image deraining using gaussian processes." in CVPR 2020.

[2]. Huaibo Huang, et al. "Memory oriented transfer learning for semi-supervised image deraining." in CVPR 2021.

[3]. Xiang Chen, et al. "Unpaired deep image deraining using dual contrastive learning." In CVPR 2022.

Reviewer comment (Experiments):
Unpaired Deep Image Deraining Using Dual Contrastive Learning, CVPR 2022, proposed a contrastive based approach to learn the representation of rain and image and guide the networks to learn a removal network and image generator networks in an unsupervised cycle-GAN approach. It would be great to see comparisons of CoI-C against this method.

Author response:
We first added the semi-supervised method Syn2Real [1], MOSS [2], and the unsupervised method DCD-GAN [3] to our *Introduction* and literature review in *Related Work*. Since the official training code for MOSS [2] is unavailable, we compare the proposed CoIC with Syn2Real and DCD-GAN. Both Syn2Real and DCD-GAN are trained for more than 1M iterations (much longer than BRN, RCDNet, DGUNet, IDT, and DRSformer). However, we observed that both Syn2Real and DCD-GAN struggle when faced with intricate rainy images from multiple datasets, yielding unsatisfactory results on Rain200L, Rain200H, Rain800, DID-Data, and DDN-Data datasets. We tabulate the PSNR metrics below for direct comparison (Details are in Table 1 of the revised manuscript):

| Methods          | Rain200L  | Rain200H  | Rain800   | DID-Data  | DDN-Data  |
| ---------------- | --------- | --------- | --------- | --------- | --------- |
| Syn2Real         | 30.83     | 17.21     | 24.85     | 26.71     | 29.15     |
| DCD-GAN          | 21.64     | 16.04     | 19.52     | 21.28     | 21.60     |
| DRSformer + CoIC | **39.81** | **30.50** | **29.92** | **35.01** | **33.94** |

From the above results, we observe that learning a comprehensive deraining model on mixed datasets is challenging for semi-supervised and unsupervised methods. The proposed CoIC can efficiently assist models in learning much better on multiple datasets.

Additionally, we have provided a demo on our anonymous GitHub page: https://anonymous.4open.science/r/CoIC-730F/

[1]. Yasarla, Rajeev, et al. "Syn2real transfer learning for image deraining using gaussian processes." in CVPR 2020.

[2]. Huaibo Huang, et al. "Memory oriented transfer learning for semi-supervised image deraining." in CVPR 2021.

[3]. Xiang Chen, et al. "Unpaired deep image deraining using dual contrastive learning." In CVPR 2022.

Reviewer comment (Evaluation):
The proposed techniques seem to be a guidance for deraining, i.e., the authors attempt to constrain an instance-specific representation to guide the deraining. Although this, the reviewer would like to know the increased parameters and FLOPs.

Author response:
As suggested by the reviewer, we have added a comprehensive analysis of CoIC in terms of the parameters (#P), FLOPs, and testing time. The results are presented in the table below. Generally, **the increased parameters are related to the intrinsic architecture of the models**. **It is noteworthy that the performance improvement is not mainly from the increased parameters**. Note that IDT equipped with CoIC, with fewer increased parameters, can result in much better performance improvement when compared to DRSformer in Table 1, where more parameters are brought by CoIC for DRSformer. However, the changes in FLOPs and inference time (about 30ms) are almost the same for all models, which means that the extra changes are mainly from the encoder but not the modulation process.

We have also included this analysis in our paper. Please refer to *Appendix A.3* for details.

| Model name | input size | #P w/o CoIC (M) | #$\Delta$P  (M) | FLOPs w/o CoIC (G) | FLOPs w/CoIC (G) | Time w/o CoIC (ms) | Time w/ CoIC (ms) |
| ---------- | ---------- | :-------------: | --------------- | ------------------ | ---------------- | ------------------ | ----------------- |
| BRN        | 512x512    |      0.38       | 0.21            | 392.9              | 393.3            | 332.2$\pm$38.1     | 364.2$\pm$33.0    |
| RCDNet     | 512x512    |      2.98       | 2.11            | 389.0              | 389.5            | 351.6$\pm$0.2      | 384.8$\pm$2.2     |
| DGUNet     | 512x512    |      3.63       | 1.62            | 396.8              | 397.2            | 161.0$\pm$5.5      | 198.8$\pm$6.8     |
| IDT        | 128x128    |      16.42      | 2.53            | 7.3                | 7.6              | 59.8$\pm$2.7       | 83.2$\pm$10.5     |
| DRSformer  | 512x512    |      33.67      | 14.12           | 440.8              | 441.1            | 833.6$\pm$0.5      | 886.7$\pm$11.0    |

Reviewer comment (Experiments):
Whether the training iterations of the proposed method are the same with the applied method? i.e., BRN, RCDNet, DGUNet, IDT, DRSformer. If not, the comparisons are unfair.

Author response:
We apologize for missing the training details. In fact, to ensure a fair comparison, all the applied models were trained with consistent iterations and the pixel-fidelity loss. The training details are provided in *Appendix A.5*.

Reviewer comment (Evaluation):
Deraining methods with high PSNR and SSIM usually tend to overfit to synthetic datasets. Hence, deraining performance would be worse on real datasets. Whether the authors can further solve this problem?

Author response:
Current models usually struggle to perfectly derain real-world images when being trained on synthetic datasets. Thanks to the valuable comments above and from other reviewers, we find it simple yet effective to achieve **remarkable** real-world deraining ability by **adding a real-world dataset to tune the model**. Specifically, we tune DRSformer on the real-world dataset SPAData for about 105k iterations, and we have observed **significant real-world deraining performance improvement**. Please see our updated visualizations in Figure 4 and Figure 14. For convenience, we strongly recommend the reviewer to check our anonymous GitHub page on: https://anonymous.4open.science/r/CoIC-730F/.

Reviewer comment (Experiments):
Can the rain-aware negative exemplars be images with real rain streaks? Whether the authors consider to explore the real images to participate in training to improve the generalization to real scenes?

Author response:
There is no limitation on the rain-aware negatives. In practice, we tune the DRSformer on both synthetic and real-world SPA datasets for about 105k iterations, where synthetic and real-world rain streaks can fully interact through Equation 3. Surprisingly, we have attained remarkable real-world deraining ability on the SPAData dataset, which demonstrates the potential of the proposed method. Therefore, **the rain-aware negative exemplars can also be real rain streaks**.

Reviewer comment (Experiments):
I have observed that the authors mention the generalization many times. However, only training on synthetic dataset is hard to improve the generalization because of overfitting.

Author response:
To enhance the *real-world generalization ability*, we fine-tune the pre-trained DRSformer on the real-world SPA dataset. Experimental results demonstrate a significant improvement in real-world deraining ability. **In summary, the proposed CoIC enables us to train a powerful model capable of effectively deraining both synthetic and real-world rain scenarios simultaneously.**