From: NeurIPS.cc/2022/Conference/Program_Chairs
title: Paper Decision
decision: Accept

From: NeurIPS.cc/2022/Conference/Paper172/Area_Chair_fAha
metareview: The paper has received positive reviews. There was substantial discussion, and the authors are strongly encouraged to include the clarifications they made rot the final copy, as well as a more extensive discussion of the limitations and possible danger of overfitting.

recommendation: Accept
confidence: Certain
award: No

From: NeurIPS.cc/2022/Conference/Paper172/Authors
title: Thanks for your new comment
comment: Thanks for your new comment again!

**Is there any intuitive reason for the proposed linear blending formulation of K in EQ7? How does the K_o component (with/without K_o) impact the performance? Have you ever tried other formulations for K?**

Though Eq. 7 is a linear blending form of kernels, the learning of kernels is done by using a set of convolutional layers and non-linear activation functions (ReLu). They form a non-linear function for adjusting the weights of the kernels, which allows the overall result of Eq. 7 to be computed by the non-linear function as well. Please also note that this manner has been widely used in the current form of CNN, where the convolutional block generally consists of consecutive linear (convolutional layer, fully-connected layer) and non-linear functions (ReLu, Sigmoid) for fitting more universal functions.

Due to the limited time for making our paper more focused, we have not experimented with other formulations of blending kernels. Yet, it should be investigated that the specific blending may be finally equivalent to using the trivial linear and non-linear functions for fitting the blending form, while the specific blending may requires extra computation and supervision.

From: NeurIPS.cc/2022/Conference/Paper172/Authors
title: Response to New Comment
comment: Thanks for your new comments!

**1. The proposed method significantly increased the number of CNN parameters in order to estimate the latent components R and K (to achieve the SOTA performance).**

During the network training, SEIDNet consists of 132 convolutional layers (26 layers for extracting the visual feature map + 27 layers for computing the reference kernels + 32 layers for constructing 2 CVAEs with the complete encoder, condition, and decoder branches + 47 layers for others) with 3.97M learnable parameters. This network complexity is reasonable and comparable to recent methods like PReNet, MPR, SPDNet, and RCDNet, which require 3~30M parameters.

During the network testing, the network complexity can be further reduced, because we can remove the convolutional layers for computing the reference kernels and constructing the encoders of CVAEs. In this case, the network consists of 90 layers (26 layers for extracting the visual feature map + 20 layers for constructing 2 CVAEs with the condition and decoder branches + 44 layers for others) with 2.5M learnable parameters.

**2. According to the formulations, the reconstruction loss L_de is the only strong constraint used to supervise the learning process. There is no ground truth of K and R in L_se (Eq5) and L_id(EQ9), or any constraints on K and R. Actually, there are infinite decoupling solutions of K, and R. Therefore, it requires more layers to increase the learning capacity of the net, and more data to learn how to correctly decouple K, and R. As a result, the proposed method is highly dependent on its training data, and much easier to get overfitted. Constraints on K, and R may be required to make the learning process more efficient and robust.**

Please note that overfitting means a method trained on a data set produces ugly results on different test sets. In Table 4 of the supplementary file, we have experimented with training our model on the unified training set of Rain13K and tested on the separate test sets (i.e., the test sets of Test100, Test1200 [10], Test2800 [11], Rain100H, and Rain100L). We remark that this is a conventional way to justify the generalization (overfitting) of the deraining model. In this case, our model still outperforms other methods. Thus, we find no evidence of overfitting of our model. 

We agree with your opinion on using stronger constraints on the status and kernel to make the learning process more efficient and robust. Actually, in Eq. 3, we subtract the object layer from the input image to achieve a relatively strong constraint on the status. Yet, how to achieve strong supervision for the kernel is difficult, because a kernel is generally regarded as a latent variable. Thus, there is no strong supervision for learning the kernel for deraining or other related tasks in the existing methods.

From: NeurIPS.cc/2022/Conference/Paper172/Reviewer_aYKW
title: Question for the equation of the information decoupling
comment: Is there any intutive reason for the proposed linear blending forumlation of K in EQ7?  How does the K_o component (with/without K_o) impact the performance?  Have you ever tried other formulations for K?

From: NeurIPS.cc/2022/Conference/Paper172/Reviewer_aYKW
title: Limitations of the proposed method
comment: The proposed method significantly increased the number of CNN parameters in order to estimate the latent components R and K (to achieve the SOTA performance).  According to the formulations, the reconstruction loss L_de is the only strong constraint used to supervise the learning process. There is no ground truth of K and R in L_se (Eq5) and L_id(EQ9), or any constraints on K and R. Actually, there are infinite decoupling solutions of K, and R. Therefore, it requires more layers to increase the learning capacity of the net, and more data to learn how to correctly decouple K, and R. As a result, the proposed method is highly dependent on its training data, and much easier to get overfitted. Constraints on K, and R may be required to make the learning process more efficient and robust.

From: NeurIPS.cc/2022/Conference/Paper172/Authors
title: Response to Reviewer CNK5
comment: Thanks for the new questions.

**1. Is this CVAE-based kernel sampling completely new or there is previous related work?**

In areas of rain/snow/shadow/haze removal and the related image restoration tasks, we find that the previous methods based on the generative networks (e.g., GAN and VAE) generally construct the latent space for sampling the image patches. Thus, they lose the chance to use kernels, which provide more explicit and accurate control of the pixel intensities. Along another line, the discriminative networks compute the shared or dynamic kernels for processing the pixel intensities. Yet, the kernels are deterministic. They yield unsatisfactory results on the confusing rain/object pixels.

Based on the discussion above and to the best of our survey, we believe that our work is novel in the rain/snow/shadow/haze removal and the related tasks, in terms of using the generative power for computing the accurate kernels. However, due to the limited time and effort, we are not sure if there are works in other vision tasks sharing a similar spirit. Thus. we are cautious in using terms like “completely new” in the paper. We are pleased if more references are recommended for discussion.

**2. Is it necessary to use networks with many layers here? What is the performance of the proposed method using the same/equivalent backbone as the previous methods? I just want to make sure the performance gain here is not simply because we are using a better backbone.**

During the network training, SEIDNet consists of 132 convolutional layers (26 layers for extracting the visual feature map + 27 layers for computing the reference kernels + 32 layers for constructing 2 CVAEs with the complete encoder, condition, and decoder branches + 47 layers for others) with 3.97M learnable parameters. This network complexity is reasonable and comparable to recent methods like PReNet, MPR, SPDNet, and RCDNet, which require 3~30M parameters.

During the network testing, the network complexity can be further reduced, because we can remove the convolutional layers for computing the reference kernels and constructing the encoders of CVAEs. In this case, the network consists of 90 layers (26 layers for extracting the visual feature map + 20 layers for constructing 2 CVAEs with the condition and decoder branches + 44 layers for others) with 2.5M optimized parameters.

**3.The result of SP+M-Net [TPAMI 22 Le et al].**

Thanks. We have added it to the tables below.

| ||&emsp; &nbsp; ISTD+||
|:-:|:-:|:-:|:-:|
|Method|Shadow|Non-shadow|All|
|PMDNet|9.7|3.0|4.0|
|AEFNet|6.5|3.8|4.2|
|CRFormer|5.9|2.9|3.4|
|SP+M-Net|9.7|3.0|4.0|
|SEIDNet|6.5|3.4|3.9|

| ||&emsp; &ensp; ISTD||
|:-:|:-:|:-:|:-:|
|Method|Shadow|Non-shadow|All|
|AEFNet|7.77|5.56|5.92|
|CRFormer|7.32|5.82|6.07|
|SP+M-Net|6.0|3.1|3.6|
|SEIDNet|7.47|5.08|5.47|

From: NeurIPS.cc/2022/Conference/Paper172/Reviewer_CNK5
title: How important is the network architecture?
comment: Thank you for the clarification! I will check the paper again for the details. 

Several following up questions:

1) Is this CVAE-based kernel sampling completely new or there is previous related work? 

2) Is it necessary to use networks with many layers here? What is the performance of the proposed method using the same/equivalent backbone as of the previous methods? I just want to make sure the performance gain here is not simply because we are using a better backbone.

Ref that you requested (sorry for not being clear in the beginning):
TPAMI-22 - Le & Samaras: Physics-based Shadow Image Decomposition for Shadow Removal
DOI Bookmark: 10.1109/TPAMI.2021.3124934

From: NeurIPS.cc/2022/Conference/Paper172/Authors
title: Response to Reviewer CNK5
comment: We are sorry for the late reply. It takes time to prepare the extensive analysis for better solving the questions.

**1.Your motivation here perfectly justifies the need for having various potential statuses per pixel among which we hopefully have one correct status.**

Actually, by sampling multiple statuses, we hopefully have more correct statuses, which help to suppress the negative impact of the incorrect statuses.

**2.However, what puzzles me the most is that eventually the proposed method samples several potential statuses and takes the average of them. The average of all sampled statuses would never be a good approximation of the correct status. The logic here is unclear to me and at least should be listed as a limitation.**

We respectfully clarify that the sampled statuses are not averaged. We agree that the average of the sampled statuses is not a good approximation of the correct status, because the average status is very sensitive to incorrect sampled statuses. This strategy involves problematic status information at the very beginning. It heavily degrades the reliability of the kernel that is estimated later. In Section 5.2 “Analysis of Network Components” of the paper, we have experimented with averaging the sampled statuses to estimate the kernel, where the performance is worse than our full model. Please see the results in the second row of Table 1. The network architecture is illustrated in Figure 1(b) of the supplementary file.

**3.Should it be more logical to first, sample various statuses, and then have a mechanism to identify the best one among these sampled kernels?**

We agree with your opinion. Instead of averaging the sampled statuses, we input the statuses, individually, into the second CVAE (see Figure 3), for sampling multiple kernels separately. Thus, the incorrect kernels have little impact on the correct ones. Please also note that the sampled kernels are computed based on the corresponding latent vectors (see $\mu_c^m + \sigma_c^m * Z^m$ at the bottom of Eq. 12). Each latent vector is processed by the convolutional weights of the decoder in the second CVAE. The latent vectors associated with the correct/incorrect kernels can be enhanced/suppressed by the convolutional weights. Though we finally average the sampled kernels, the incorrect kernels, which are computed based on the suppressed latent vectors, have little impact on the average kernel. In Section 5.2 “Sensitivity to the Number of Kernels”, we have evaluated the effectiveness of sampling more kernels for deraining.

To further justify the effectiveness of the average kernel, we have added an extensive analysis to Section 9 “Analysis of the Average Kernels” of the supplementary file. We manually select 100 pairs of image patches from Rain100L and Rain1400 (see Figure 16(a) of supplementary file). Each pair of image patches contain the rain streaks and the object textures, respectively, where the rain streaks and the object textures are visually similar. The typical discriminative networks (i.e., EfDeRain, SPDNet, and MPR) compute similar kernels for the confusing rain and object, thus yielding unsatisfactory results. In contrast, we sample more kernels for each image patch, where the sampled kernels are averaged. We compute the difference (L1 distance) between the average kernels, which belong to each pair of confusing rain and object patches. We accumulate and average the differences, which are reported in Figure 16(b) of supplementary file. With more sampled kernels, we achieve more differentiable average kernels for processing the confusing rain and object patches.

**4.Furthermore, I think the method, even with the VAE, only looks at a small local patch to estimate the status. Without more contexts, how does it conceptually resolve the status uncertainty?**

We agree that the multi-scale context is critical to estimating the correct status and kernel. This can be partially evidenced in our discussion on the limitation of our method (see Section 5 of the supplementary file). Fewer network layers require less computation. However, they achieve less global context for computing the accurate kernels, eventually degrading the performance. Thus, it is future work to incorporate our framework with the cost-effective module for learning the multi-scale context.

**5.Please add recent shadow removal methods to the table (TPAMI 22 Le et al., CRFormer Wan et al.).**

Thanks. We have added the result of CRFormer [Wan et al.] to the tables below. The results are reported on ISTD and ISTD+. We also respectfully request the detailed reference information of [TPAMI 22 Le et al.] for adding their results.
| ||&emsp; &nbsp; ISTD+||
|:-:|:-:|:-:|:-:|
|Method|Shadow|Non-shadow|All|
|PMDNet|9.7|3.0|4.0|
|AEFNet|6.5|3.8|4.2|
|CRFormer|5.9|2.9|3.4|
|SEIDNet|6.5|3.4|3.9|

| ||&emsp; &ensp; ISTD||
|:-:|:-:|:-:|:-:|
|Method|Shadow|Non-shadow|All|
|AEFNet|7.77|5.56|5.92|
|CRFormer|7.32|5.82|6.07|
|SEIDNet|7.47|5.08|5.47|

From: NeurIPS.cc/2022/Conference/Paper172/Authors
title: Sincerely Request Your New Comment
comment: Dear Reviewer 4SNd,

Thanks for your review again. As the deadline for the authors' response is approaching, we sincerely request your comment on our primary response. This will definitely give us a valuable chance to address the questions unsolved.

Best,

Authors of Paper ID 172

From: NeurIPS.cc/2022/Conference/Paper172/Authors
title: Sincerely Request Your New Comment
comment: Dear Reviewer YErN,

Thanks for your review again. As the deadline for the authors' response is approaching, we sincerely request your comment on our primary response. This will definitely give us a valuable chance to address the questions unsolved.

Best,

Authors of Paper ID 172

From: NeurIPS.cc/2022/Conference/Paper172/Authors
title: Sincerely Request Your New Comment
comment: Dear Reviewer aYKW,

Thanks for your review again. As the deadline for the authors' response is approaching, we sincerely request your comment on our primary response. This will definitely give us a valuable chance to address the questions unsolved.

Best,

Authors of Paper ID 172

From: NeurIPS.cc/2022/Conference/Paper172/Reviewer_CNK5
title: Thanks!
comment: Thank you a lot for your response. 

>But the status is not trivially a binary indicator of rain or object.

I agree with this.

>Different pixel-wise statuses should be respected by different kernels, which have different weights to suppress/enhance the rain/object intensities of the corresponding pixels.

I agree with this.

Your motivation here perfectly justifies the need for having various potential statuses per pixel among which we hopefully have one correct status (please correct me if I am wrong here). However, what puzzles me the most is that eventually the proposed method samples several potential statuses and takes the average of them. I don't get the logic here. To me, the average of all sampled statuses would never be a good approximation of the correct status. It surely is better than the worst but can't be the best. The logic here is unclear to me and at least should be listed as a limitation. Should it be more logical to first, sample various statuses, and then have a mechanism to identify the best one among these sampled kernels? Furthermore, I think the method, even with the VAE, only looks at a small local patch to estimate the status. Without more contexts, how does it conceptually resolve the status uncertainty?  

This is the only significant issue I have with this paper and I think it is quite important. Without proper explanation here, I don't understand why the method works. 

>Experiment on the Adjusted ISTD dataset.

Thanks! I see that the method achieves quite competitive results here but not SOTA (which is fine in my opinion since the method isn't designed for this task specifically). Please add recent shadow removal methods to the table (TPAMI 22 Le et al., CRFormer Wan et al.)

From: NeurIPS.cc/2022/Conference/Paper172/Authors
title: Response to Reviewer 4SNd
comment: Dear Reviewer 4SNd,

We sincerely thank you for your constructive comments for making our paper much better! We are looking forward to your comments on our response. Should you have any other questions, we will do our best to response to you.

Best,

Authors of Paper ID 172

From: NeurIPS.cc/2022/Conference/Paper172/Authors
title: Response to Reviewer CNK5
comment: Dear Reviewer CNK5,

We sincerely thank you for your constructive comments for making our paper much better! We are looking forward to your comments on our response. Should you have any other questions, we will do our best to response to you.

Best,

Authors of Paper ID 172

From: NeurIPS.cc/2022/Conference/Paper172/Authors
title: Response to Reviewer YErN
comment: Dear Reviewer YErN,

We sincerely thank you for your constructive comments for making our paper much better! We are looking forward to your comments on our response. Should you have any other questions, we will do our best to response to you.

Best,

Authors of Paper ID 172

From: NeurIPS.cc/2022/Conference/Paper172/Authors
title: Response to Reviewer aYKW
comment: Dear Reviewer aYKW,

We sincerely thank you for your constructive comments for making our paper much better! We are looking forward to your comments on our response. Should you have any other questions, we will do our best to response to you.

Best,

Authors of Paper ID 172

From: NeurIPS.cc/2022/Conference/Paper172/Authors
title: Response to Reviewer aYKW
comment: **1. Explanation of “conv” in Eq. 3, Eq. 6, and Figure 2.**

In Eq. 3, “conv” is a convolutional layer. In Eq. 6 and Figure 2(d), “conv” means a set of convolutional layers. The kernel size is 3x3. We eliminate the notations of the convolutional kernels to simplify the variables in the paper.

**2. Explanation of encoder, condition, and decoder.**

The encoder, condition, and decoder branches are convolutional blocks. We have introduced the details of the encoder, condition, and decoder branches in Section 1 of the supplementary file. Each branch has 16 convolutional layers with 3x3 kernels. There is a pair of batch normalization and ReLU layers between the adjacent convolutional layers.

**3. Explanation of $condition(F)$ and $condition(F, R)$ for computing the mean values and standard deviations.**

$condition$ in Eqs. 4 and 12(top) is the condition branch of the CVAE $\mathcal{V}_{se}$. $condition(F)$ means that we pass the feature map F to the convolutional layers of the condition branch. The last convolutional layer outputs the mean value and standard deviation maps, which helps to sample multiple status maps from the latent space.

$condition$ in Eqs. 8 and 12(bottom) is the condition branch of the CVAE $\mathcal{V}_{id}$. $condition(F, R)$ means that we concatenate the feature map F and the status map R, then feed them into the convolutional layers of the condition branch. The last convolutional layer outputs the mean value and standard deviation maps, which are used to sample multiple kernel maps from the latent space.

**4. Explain the equation of the information decoupling.**

The information decoupling is formulated as Eqs. 6 and 7. First, we achieve the element-wise multiplication between the feature map F and the status map R, yielding the feature map $F_r$ of the rain streaks in the image. We also compute the element-wise multiplication between the feature map F and the reverse status map (1-R), achieving the feature map $F_o$ of the object textures. Next, $F_r$ and $F_o$ are fed into different convolutional blocks (denoted as “conv”), which yield the kernel maps $K_r$ and $K_o$. The kernel maps $K_r$ and $K_o$ are weighted by the status map R (see Eq. 7). The weighted kernel map K is used to remove/enhance the rain/object intensities of the image.

**5. How to get the correct kernel map K?**

Before training the CVAE $\mathcal{V}_{id}$, we pre-train the convolutional layers in Eq. 6. In this manner, the computation of the kernel map is directly supervised by the deraining loss. Thus, the kernel map can play as an accurate reference for training this CVAE.

**6. Explanation of the discriminative networks.**

Note that the architectures of the discriminative networks in the ablation study have been illustrated in Figure 1 (a-c) and Figure 2 (a,c) of the supplementary file. We will release the implementation of these discriminative networks along with the code package.

**7. Limitation of the proposed method.**

We have discussed the limitation in Section 5 of the supplementary file. We change the number of convolutional layers (see “conv” in Eq. 6) to compute the kernel map K (see Eq. 7), which plays as a reference for training the CVAE $\mathcal{V}_{id}$. Though the pre-training of the convolutional layers for achieving the kernel map K is supervised by the deraining loss, more layers show a strong power for estimating a more accurate kernel map K, however, at the cost of more computation (see Figure 11(a-b)). Compared to the extremely fast methods (e.g., EfDeRain), SEIDNet still has much room to achieve a better trade-off between performance and computational efficiency.

**8. Comparison of the proposed CVAE with GAN or other discriminative methods.**

Thanks. In Table 4, we have compared SEIDNet with the generative CVID and other discriminative methods on the deraining task. An array of GAN-based methods have been compared on the desnow and deshadow tasks. Among these methods, Composition GAN, DS-GAN, and RFMPRaLSGAN are used for desnow; Mask-GAN, ARGAN, and RIS-GAN are used for deshadow. We have added the results of the recent GAN-based methods (i.e., DCD-GAN and DerainCycleGAN) on the deraining task, and the results of DW-GAN and FD-GAN on the dehaze task in the following tables.

Please note that our major contribution is to model the distributions of the pixel-wise status and kernel. CVAE is an implementation to learn these distributions, whose effectiveness has been evidenced in the ablation study and state-of-the-art comparison. In the future, we will study the implementation with stronger generative networks (i.e,CGAN, CVAE-GAN, Transformer-based CVAE), which hopefully improve the performance.

| | &emsp; &ensp;SPA |
|:-:|:-:|
|Method|PSNR &nbsp; SSIM|
|DerainCycleGAN|35.20 &nbsp; 0.9500|
|DCD-GAN |35.30 &nbsp; 0.9430|
|SEIDNet|44.96 &nbsp; 0.9911|

| |&nbsp; ITS Subset|
|:-:|:-:|
|Method|PSNR SSIM|
|FD-GAN|23.15 0.9207|
|DW-GAN|35.94 0.9860|
|SEIDNet|40.62 0.9968|

From: NeurIPS.cc/2022/Conference/Paper172/Authors
title: Many thanks to all reviewers for their constructive comments
comment: We sincerely thank the reviewers for their constructive comments. We are pleased to see that our contribution is unanimously found novel the reviewers. Below, we address the concerns raised by the reviewers.

From: NeurIPS.cc/2022/Conference/Paper172/Authors
title: Response to Reviewer 4SNd
comment: **1. Errors on some bright objects.**

Thanks for your valuable comment, which helps us to better clarify the analysis in the supplementary file.

It should be noted that bright objects are extremely similar to the appearances of the rain streaks. The examples of the confusing bright objects and rain streaks can be found in Figure 6: (1) the top-right water region in the first row; (2) the second-left person in white cloth in the fourth row; (3) the bright regions of the wheels in the last row. There are pixels of these bright objects misunderstood as rain streaks, as illustrated in the corresponding status maps. Yet, we use multiple status maps, which provide more differentiable information for separating the bright objects from the rain streaks. Thus, the final deraining results are reasonable. In future work, we plan to further improve the deraining results of the bright objects, while relying on fewer status maps for saving computation.


**2. Discussion on size of training and testing dataset.**

We sincerely thank you for this constructive comment, which helps to make our experimental analysis more comprehensive.

Please note that public datasets are used for a fair comparison between SEIDNet and other methods. Some of the datasets contain relatively balanced training and testing splits, e.g., Snow100K (50K/50K images for training/testing), Rain100L (200/100 images for training/testing), and ISTD dataset (1330/540 images for training/testing). In these datasets, SEIDNet outperforms other methods.

We agree that conducting experiments on the unbalanced training and testing splits may reduce the confidence of SEIDNet. Thus, we follow your suggestion and reduce the training data in the extremely unbalanced dataset (i.e., SPA with 638K/1K images for training and testing). This is done by randomly sampling 1K training images from SPA dataset. The random sample is three-fold, thus forming three different subsets, each of which contains 1K images for training different models. The trained models are evaluated on the 1K images in the original test set. With different subsets for training, SEIDNet yields better results than other methods (see the table below).


 ||&emsp; Subset A|&emsp; Subset B|&emsp; Subset C|
|:-:|:-:|:-:|:-:|
|Method|PSNR &ensp; SSIM|PSNR &ensp; SSIM|PSNR &ensp; SSIM|
|EfeDeRain|&nbsp; 34.37 &nbsp; 0.9556|&nbsp; 36.18 &nbsp;  0.9629|&nbsp; 35.44 &nbsp; 0.9576|
|MPR|&nbsp; 35.57 &nbsp; 0.9519|&nbsp; 36.83 &nbsp; 0.9579|&nbsp; 37.65 &nbsp; 0.9630|
|SPDNet|&nbsp; 31.84 &nbsp; 0.9094|&nbsp; 33.57 &nbsp; 0.9254|&nbsp; 32.93 &nbsp; 0.9162|
|SEIDNet|&nbsp; 39.07 &nbsp; 0.9813|&nbsp; 39.90 &nbsp; 0.9821|&nbsp; 39.97 &nbsp; 0.9799|


**3. No societal impact.**

Many thanks for your reminder. Our approach can help to recover the image information, which may be broadly used in many scenarios (e.g., autonomous vehicles and video surveillance). One should be cautious of using the results, which may contain problematic information. This may give rise to the infringement of privacy or economic interest. We have added this societal impact to Section 6 of the supplementary file.

From: NeurIPS.cc/2022/Conference/Paper172/Authors
title: Response to Reviewer CNK5
comment: **1. How to connect the motivation to the proposed method? What does it mean to randomly sample multiple status maps given a single input image? Why are multiple statuses (kernels) better than a single status (kernel), in terms of resolving the confusing information of the pixels?**

We sincerely thank you for this valuable comment, which significantly helps us to clarify the motivation for proposing SEIDNet and how multiple statuses and kernels work. We will also follow your suggestion to polish the presentation of our paper.

$[Motivation$ $of$ $Proposing$ $SEIDNet]$

In the input image, the observed intensity of each pixel can be regarded as a mixture of the pixel intensities of rain and object. We agree that each pixel is associated with an underlying status. But the status is not trivially a binary indicator of rain or object. It should be a factor that determines the rain and object intensities. Different pixel-wise statuses should be respected by different kernels, which have different weights to suppress/enhance the rain/object intensities of the corresponding pixels.

Typically, the discriminative network outputs a single status for each pixel. As introduced in lines 24-28 of the paper, the rain streaks and object textures may be similar (see the example in Figure 1). They mislead the discriminative network (e.g., EfDeRain), which outputs similar statuses for the similar rain and object pixels with confusing appearances. In this case, the similar rain and object pixels cannot be differentiated. The similar statuses let the corresponding pixels be processed by similar kernels, yielding erroneous deraining results. This problem motivates us to propose SEIDNet for using multiple statuses and kernels to better differentiate and process the confusing rain and object pixels.

$[The$ $concept$ $of$ $sampling$ $multiple$ $statuses$ $and$ $why$ $it$ $work]$

We use the first CVAE of SEIDNet to sample multiple statuses for each pixel. Given the rain and object pixels in different training images, this CVAE learns the distribution of the pixel-wise statuses. Conceptually, the distribution can be regarded as a latent space, where we embed the statuses of the rain and object pixels of the training images. To infer the status of an unknown pixel, we sample from the latent space to find multiple possible statuses, which appear in similar scenarios of the training images. Rather than using an indifferentiable status alone, we resort to multiple statuses together to borrow more differentiable information from a broad range of training images, thus providing a better chance to separate the confusing pixel from the similar pixels.

$[The$ $concept$ $of$ $sampling$ $multiple$ $kernels$ $and$ $why$ $it$ $work]$

The second CVAE of SEIDNet constructs the second latent space. In this space, we embed the kernels for processing the rain and object pixels in the training images. By respecting multiple possible statuses of the pixel, we again borrow the information widely from the training images, sampling multiple kernels from the second latent space. We average these kernels to process the corresponding pixel. Given a pair of confusing rain and object pixels, their average kernels likely have different weights to adjust their rain and object intensities, compared to the single and similar kernels produced by the discriminative network. The effectiveness of using multiple kernels has been evidenced by the experiment in Section 5.2 “Sensitivity to the Number of Kernels” of the paper.

**2. Experiment on the Adjusted ISTD dataset.**

Thanks for your suggestion. In the table below, we compare SEIDNet with the recent methods PMDNet [ECCV 2020] and AEFNet [CVPR 2021] on the Adjusted ISTD dataset (a.k.a., ISTD+). SEIDNet outperforms the compared methods.

| ||&emsp; ISTD+||
|:-:|:-:|:-:|:-:|
|Method|Shadow|Non-shadow|All|
|PMDNet|9.7|3.0|4.0|
|AEFNet|6.5|3.8|4.2|
|SEIDNet|6.4|3.4|3.9|

**3. Limitation of when the method doesn't work well in comparison to other approaches and why.**

Thanks. We have discussed the limitation of when SEIDNet works unsatisfactorily (see Section 5 of the supplementary file). Here, we further clarify this limitation by comparing SEIDNet with other methods.

The major limitation of SEIDNet stems from the need for a deeper network with 27 convolutional layers to estimate the kernels, which play as the reference for training the second CVAE satisfactorily. Currently, using these extra 27 layers significantly lowers the inference speed of SEIDNet, which requires 0.083s to process an image. Compared to the extremely fast models like EfeDerain (0.0059s/image), SEIDNet stills has much room to achieve a better trade-off between performance and computational efficiency.

From: NeurIPS.cc/2022/Conference/Paper172/Authors
title: Response to Reviewer YErN
comment: **1. Experiments on multiple degradations.**

Thank you for this valuable comment. According to your suggestion, we compare our SEIDNet with the advanced methods (EfeDeRain [AAAI 2021],HRGAN [CVPR 2019] MPR [CVPR 2021], TransWeather[CVPR 2022]) on the Outdoor-Rain Test1 dataset [CVPR 2019], where the images are degraded by rain and fog. We report the results of different methods in the following table, where SEIDNet outperforms other methods in terms of PSNR and SSIM.

| |&emsp;&ensp; Test 1&emsp;  |
|:-:|:-:|
|Method|PSNR &emsp; SSIM|
|MPR|21.90 &ensp; 0.8456|
|HRGAN|21.56 &ensp; 0.8550|
|EfeDerain|22.96 &ensp; 0.8842|
|TransWeather|31.05 &ensp; 0.9509|
|SEIDNet|31.36 &ensp; 0.9593|

**2. Comparisons with recent methods (Maxim [CVPR 2022] and Transweather [CVPR 2022]).**

Many thanks for your suggestion. In comparison with the contemporary methods (Maxim [CVPR 2022] and Transweather [CVPR 2022]), SEIDNet still shows its effectiveness by achieving better results on the challenging deraining (Rain13K), dehazing (ITS\&OTS), desnow (Snow 100K), and multi-degradation (Outdoor-Rain Test1) datasets. Please see the tables below.

| |Snow100K Overall|
|:-:|:-:|
|Method|&ensp; PSNR &emsp; SSIM&ensp; &nbsp; |
|TransWeather|31.55 &ensp; 0.9152|
|SEIDNet|32.77 &ensp; 0.9643|

| |&nbsp; &nbsp; ITS Subset &nbsp;|&nbsp; &nbsp;OTS Subset|
|:-:|:-:|:-:|
|Method|PSNR &ensp; SSIM|PSNR &ensp; SSIM|
|MAXIM|&nbsp; 38.11 &nbsp; 0.9910|&nbsp; 34.19 &nbsp; 0.9850|
|SEIDNet|&nbsp; 40.62 &nbsp; 0.9968|&nbsp; 35.72 &nbsp; 0.9951|

| |Rain13K Overall|
|:-:|:-:|
|Method|PSNR &emsp; SSIM|
|TransWeather|&nbsp; 31.48 &ensp;&nbsp; 0.9104|
|MAXIM|&nbsp; 33.37 &ensp;&nbsp; 0.9365|
|SEIDNet|&nbsp; 33.62 &ensp;&nbsp; 0.9539|

**3. Make code and pseudo-code public.**

We will definitely follow your suggestion and release the code package and pseudo-code publicly. For your reference, we have provided some of the core code segments of SEIDNet in the supplementary file. We have also attached the below pseudo-code for training and testing to Section 7 of the supplementary file. 

$Algorithm$ $A$: Training pseudo-code of SEIDNet

1: $epoch$ = 1;

2: while $epoch$≤ max_epoch do 

3: &emsp;  Input rainy image $I$, object layer $O$ for estimating status map $R$ via Eq. (3);
    
4: &emsp;  Extract feature map $F$ from rainy image $I$;

5: &emsp; Pass feature map $F$ and status map $R$ to CVAE $\mathcal{V}_{se}$;
    
6: &emsp; Estimate mean values and standard deviation maps: $[μ_r,σ_r] \leftarrow encoder([F, R]), [μ_f,σ_f]←condition(F)$ in Eq. (4);

7: &emsp; Input $F, Z$ and $(μ_r , σ_r )$ to the decoder of $\mathcal{V}_{se}$ to generate $R^′$ via Eq. (4);

8: &emsp; Calculate status estimation loss $L_{se}$ via Eq. (5);

9: &emsp; Estimate kernel maps $K_r$ and $K_o$ from $F$, $R$ via Eq. (6);

10:&emsp; Estimate kernel map:$K←R⊙K_r +(1−R)⊙K_o$ in Eq.(7)

11:&emsp; Pass $K$, $F$ and $R$ to CVAE $\mathcal{V}_{id}$;

12:&emsp; Calculate mean value maps and standard deviation maps$[μ_k,σ_k]←encoder([K, F, R]), [μ_c,σ_c]←condition([F,R])$ in Eq. (8);

13:&emsp; Generate kernel map from kernel space: $K^′←decoder([F,R,μ_k+σ_k⊙Z])$ in Eq. (8);

14:&emsp; Calculate information decoupling loss $L_{id}$ via Eq. (9);

15:&emsp; Employ $K^′$ and $I$ to estimate object layer $O^′$ via Eq. (10);

16:&emsp; Calculate deraining loss $L_{de}$ via Eq. (10);

17:&emsp; Calculate overall loss $L$ via Eq. (11);

18:&emsp; Update network weights;

19:&emsp;  If $epoch$ ≥ lr_decrease_epoch then

20:&emsp;  &emsp; Adjust learning rate;

21:&emsp; end if

22:&emsp; $epoch \leftarrow epoch + 1$;

23: end while

$Algorithm$ $B$: Testing pseudo-code of SEIDNet

1: $index$ = 1

2: while $index$ ≤ len(test_dataset) do

3: &emsp; Extract feature map $F$ from rainy image $I$;

4: &emsp; Pass feature map $F$ to CVAE $\mathcal{V}_{se}$;

5: &emsp; Calculate mean value map and standard deviation map: $[μ_f,σ_f ]←condition(F)$ in Eq. (12);

6: &emsp; while $m$ ≤ $N$ do

7: &emsp; &emsp; Generate status map from constructed status space: $R^m←decoder([F,μ_f+σ_f⊙Z^m])$ in Eq. (12);

8: &emsp; &emsp; Pass $F$, $R^m$ and $Z^m$ to CVAE $\mathcal{V}_{id}$;

9: &emsp; &emsp; Calculate mean value map and standard deviation map: $μ^m_c,σ_c^m←condition([F,R^m])$ in Eq. (12);

10: &emsp;&emsp;Generate kernel map from constructed kernel space:$K^m←decoder([F,R^m,μ^m_c+σ_c^m⊙Z^m])$ in Eq. (12);

11: &emsp;&emsp;$m←m+1$

12: &emsp;end while

13: &emsp;Estimate $K^u$ via Eq. (13)

14: &emsp; $O$ = $K^u \circledast I$

15: &emsp; $index←index+1$

16: end while

From: NeurIPS.cc/2022/Conference/Paper172/Reviewer_aYKW
rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
confidence: 5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.
summary: The paper presents a CVAE(conditional variational auto-encoder) based method of single image rain removal. The key idea is to use the generative network VAE to learn the probability distribution of the pixel-wise rain status P(R|F,Z) and the derainning kernels P(K|F, R, Z), where R, F is the status map and the kernels, respectively. F are the extracted features of the input image, and Z is the latent variables generated by the normal distribution P(Z). For the inference stage, they use the rainy image I to compute F, and randonly generate m Zs. By using F, m Zs, they compute m Rs and consequently m Ks. The final derain kenerl map K is the mean value map of m Ks. The experimental results demostrate the effictiveness of the proposed method on the deraining benchmarks of Rain100H, Rain100L, Rain1400, SPA, the snow removal task dataset of Snow100k, and the haze removal task dataset of ITS&OTS. 

strengths_and_weaknesses: The proposed method is an extension of [Du et.al. TIP 2020]. Instead of using CVAE to directly estimate P(Y|F,Z) as Du et. al., they formulate the problem as P(R, K| F, Z) and factorize the distribution P(R, K| F, Z) = P(K|R,F,Z) P(R|F, Z), so that they can decouple the latent factors to learn P(R|F,Z) and P(K|R, F, Z), respectively. They experimental results show the effictiveness of the proposed formulation. 

However, the paper presentation is lack of clarity. There are many places in the method(both formulation and architecture) which are not clear. Especially about information decoupling, the authors should give a more explaination about why they use the corrent formulation of K (EQ6~9), and how it can get correct K as ground truth.   
questions: Please provide the clear defination of the function "conv" in EQ3, EQ6. If it is a convolution, it should have two varibles but not one. if not, please give the clear explaination. 
Please explain the "encoder", "condition", and "decoder" operations in the paper. If they are CNN blocks, please decribe them clearly, especially where the mean and standard deviation are computed from by the "condition" block ( condition(F) in EQ4,  condition(F, R) in EQ12). 
Please explain the equations of the information decoupling. Please explain the conv function in Eq6 and fig. 2. Is the conv a fixed numarical fountion? How do you promise the current formulation get correct K of  the input pair?      

Please provide more explaination about the discriminative networks they used in the comparison experiments. No information is provided in the current draft. 
limitations: The authors have not provide discussion about the limitations or failure cases of their work. The authors should include such discussions. In ablation study, more discussions and analysis are needed about the comparison of the proposed CVAE with GAN or other discriminative methods. 
ethics_flag: No
soundness: 3 good
presentation: 2 fair
contribution: 3 good

From: NeurIPS.cc/2022/Conference/Paper172/Reviewer_YErN
rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
summary: Paper proposes an image deraining method where they compute pixel status in status space that indicates whether the pixel is belonging to rain or an object.  this information is used in the decoupling layer to construct kernel space for the pixel to derain and recover a clean background.
strengths_and_weaknesses: Weaknesses:
- Can the proposed method handle multiple degradations like images with rain and fog or images with fog and snow.
- comparison with a few recent papers is missing
[a] Tu et al., Maxim: Multi-axis mlp for image processing, cvpy 2022
[b] Valanarasu et al., Transweather: Transformer-based restoration of images degraded by adverse weather conditions, cvpr 2022.

- can authors elaborate or provide pseudo-code for training and testing
- Proposed methods seem complex to implement, will the code be made public

questions: Please refer weaknesses
limitations: Please refer weaknesses
ethics_flag: No
ethics_review_area: I don’t know
soundness: 3 good
presentation: 3 good
contribution: 3 good

From: NeurIPS.cc/2022/Conference/Paper172/Reviewer_CNK5
rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
confidence: 3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
summary: The paper discusses a method for image rain removal based on two CVAE models. The first CVAE learns the distribution of rain condition (status), conditioned on the image features. The second CVAE learns the distribution of kernels that are used to remove rain effects, conditioned on the rainy status and the image features. During testing, multiple status maps and kernels are repeatedly sampled from the distribution to remove rainy effects from the input image. 
strengths_and_weaknesses: Strengths:
The technical details are clear. The idea of using CVAE to learn the distribution of status estimation and the corresponding restoration kernels is novel. The results are strong and the framework seems to work well for not just rain removal but also other image restoration tasks including snow, haze, and shadow removal. 

Weaknesses:

My main concern is that it is unclear to me how the method works. Conceptually, I think we aim to correctly estimate the status maps and the kernels, given the image feature? With the correct status map and kernels then we can remove rainy effects properly. However, the two CVAEs sample several status maps as well as multiple kernels, conditioned on the image features. I don’t understand how these randomly sampled status maps approximate rainy status in the input image. Should an input rainy image be associated with exactly one status map? What does it mean (conceptually) to randomly sample multiple status maps given a single input image? Moreover, I also don’t understand why taking the average of randomly sampled kernels is better than one kernel. While I am aware that the authors have provided quantitative evidence in the s.m. that using these CVAEs is better than using regression models directly, I don’t think the authors have provided proper explanations for the effectiveness of their method.

The motivation also doesn’t make much sense to me since I can’t connect the motivation to the proposed method. Why the “confusing information of the pixels” can be resolved by sampling multiple pseudo rainy status maps? 

The shadow removal experiment should be conducted on the Adjusted ISTD dataset (shadow image decomposition for shadow removal - ICCV19) since the original ISTD dataset exhibit a major color shift between input and GT images.

questions: Please see the weaknesses. 
limitations: Some limitations have been discussed in the s.m. It's unclear to me when the method doesn't work well in comparison to other approaches and why.
ethics_flag: No
soundness: 3 good
presentation: 3 good
contribution: 2 fair

From: NeurIPS.cc/2022/Conference/Paper172/Reviewer_4SNd
rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
confidence: 3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
summary: A CNN structure based on the combination of two Conditionally Variational Auto-Encoders (CVAE) is proposed for image rain removal. Two CVAE are used, one for generating hypothesis on rain presence at a pixel and on for the scale of the rain steak. The proposed structure is tested on several database and compared to several others methods. It is also applied separately to snow, haze and shadow removal.  
strengths_and_weaknesses: The proposed structure is interestingly introduced and seems original. The evaluation seems substantial and show improvements over 5 others methods. It is interesting that the proposed structure can work on other applications such as snow, haze and shadow removal.  
questions: In the supplementary material, shown maps of rainy pixels have numerous errors on bright objects. May it be possible de comment on this point ? 
One may notice that the size of the test database is small or very small compared to the size of the learning databases in several cases in the experiments. This may induce a lack of confidence on the proposed results that should be interesting to discuss ?
 
limitations: No societal impact.
ethics_flag: No
soundness: 3 good
presentation: 3 good
contribution: 2 fair