Paper: NIR-Assisted Image Denoising: A Selective Fusion Approach and A Real-World Benchmark Dataset (ICLR-2024)

Reviewer comment (Novelty):
I am concerned that the overall contributions are trivial. Especially, the proposed SFM module should be provided with more design reasons.

Author response:
We thank the reviewer for the valuable comments and suggestions. We appreciate the reviewer's questions and hope our responses can address the concerns.  

`1.` **Overall contributions.**

Previous related works perform NIR-assist image denoising on synthetic noisy images. The most important contribution of this work is to introduce this into the real world. We will revise the paper to make the contribution clear. 
 
Moreover, other main contributions can be summarized as follows:  

(1) Towards the content inconsistency issue, we propose a simple yet efficient selective fusion module (SFM) that can be seamlessly integrated into existing denoising networks with few increasing computation costs.
  
(2) Towards the real-world dataset scarcity issue, we construct a real-world NIR-assisted image denoising (NAID) dataset that covers diverse scenarios and various noise levels, which is expected to serve as a benchmark for future works.  

(3) Extensive experiments demonstrate that our method achieves better results than state-of-the-art ones.  

In addition, we promise the dataset, codes, and pre-trained models will be publicly available, which may be advantageous for future studies.

`2.` **Motivation of the SFM module.**

Color and structure inconsistency issues between NIR-RGB images are the main challenges in NIR-assisted image denoising. First, the NIR image is captured under additional NIR light and is monochromatic, which leads to color discrepancy between NIR-RGB images. Second, the NIR image may 'more-see' or 'less-see' the objects than the visible light ones, primarily due to inherent differences in the optical properties within each spectral domain, which leads to structure discrepancy between NIR-RGB images. 
 
On the one hand, we hope to design a module to address two inconsistency issues in this work. Thus, we present a global modulation module (GMM) and a local modulation module (LMM) to handle the color and structure inconsistencies, respectively. The combination of the two modules forms SFM. On the other hand, we hope the module can be simple, efficient, and plug-and-play. Thus, we intentionally avoid some complex operations (e.g., self-attention and cross-attention) during design in order to maintain efficiency as much as possible.  

In addition, in the second response of '**Response to Reviewer YVkk**', we further elaborate on the advantages and characteristics of the proposed SFM.  

We will revise the relevant descriptions to make SFM clearer in the revision.

Reviewer comment (Theory):
For GMM and LMM, NIR and RGB are concatenated along channel dimensions to input the same subsequent modules, and then split to obtain the two estimated NIR weights and RGB weights to apply to the corresponding branches, respectively. It seems to me that two estimated weights are the same, how to achieve selective fusion? Or are manually setting parameters involved? It would be better to provide some justification or motivation for the design choices.

Author response:
`3.` **How to achieve selective fusion?**

We apologize for the misunderstanding caused by the unclear description of the proposed SFM. Here we provide a more detailed and clear description. 
 
SFM consists of a GMM, an LMM, and a fusion operation, where GMM mainly handles the color discrepancy and LMM handles the structure one. It is worth noting that SFM automatically predicts the weights for RGB and NIR features, without manually setting parameters. And the selective fusion is mainly inspired by ‘Selective Kernel Networks’[a].  

Specifically, for GMM, it first concatenates NIR features $\mathbf{F_{N}} \in \mathbb{R}^{C \times H \times W}$ and RGB ones $\mathbf{F_{R}} \in \mathbb{R}^{C \times H \times W}$ as inputs, generating initial weights $\mathbf{W_{init}^{g}} \in \mathbb{R}^{2C \times H \times W}$ by a $1\times1$ convolutional layer, two $1\times1$ blocks for feature enhancement and another $1\times1$ convolutional layer. It can be written as, 
$$\mathbf{W_{init}^{g}} = \mathrm{Conv_{1\times1}}(\mathrm{Blocks_{1\times1}}(\mathrm{Conv_{1\times1}}([\mathbf{F_{N}},\mathbf{F_{R}}]))),$$

where $\mathrm{Conv_{1\times1}}$ denotes the $1\times1$ convolutional layer and $\mathrm{Blocks_{1\times1}}$ denotes two $1\times1$ blocks. Next, $\mathbf{W_{init}^{g}}$ is split along channel dimension to get NIR weight maps $\mathbf{\tilde{W} _{N}^{g}} \in \mathbb{R}^{C \times H \times W}$ and RGB ones $\mathbf{\tilde{W} _{R}^{g}} \in \mathbb{R}^{C \times H \times W}$. Then, a 
channel-wise softmax operation is applied to  $\mathbf{\tilde{W} _{N}^{g}}$ and  $\mathbf{\tilde{W} _{R}^{g}}$, getting the final NIR modulation weights $\mathbf{W}^g _\mathbf{N} \in \mathbb{R}^{C \times H \times W}$ and RGB ones $\mathbf{W}^g _\mathbf{R} \in \mathbb{R}^{C \times H \times W}$, which can be written as,
$$ [\mathbf{W}^g _\mathbf{N}] _{i} = \frac{\exp([\mathbf{\tilde{W}}^g _\mathbf{N}] _{i})}{\exp([\mathbf{\tilde{W}}^g _\mathbf{N}] _{i}) + \exp([\mathbf{\tilde{W}}^g _\mathbf{R}] _{i})},  \qquad[\mathbf{W}^g _\mathbf{R}] _{i} = \frac{\exp([\mathbf{\tilde{W}}^g _\mathbf{R}] _{i})}{\exp([\mathbf{\tilde{W}}^g _\mathbf{N}] _{i}) + \exp([\mathbf{\tilde{W}}^g _\mathbf{R}] _{i})}.$$

where $[\ \cdot\ ] _{i}$ denotes the selection of the $i$-th channel. Finally, we modulate the NIR features $\mathbf{F} _\mathbf{N}$ and the RGB ones $\mathbf{F} _\mathbf{R}$ with $\mathbf{W}^g _\mathbf{N}$ and $\mathbf{W}^g _\mathbf{R}$, respectively, i.e.,
$$
\mathbf{F}^g _\mathbf{N} = \mathbf{W}^g _\mathbf{N} \odot \mathbf{F _{N}},
\quad
\mathbf{F}^g _\mathbf{R} = \mathbf{W}^g _\mathbf{R} \odot \mathbf{F _{R}}, 
$$

where $\mathbf{F}^g _\mathbf{N} \in \mathbb{R}^{C \times H \times W}$ and $\mathbf{F}^g _\mathbf{R} \in \mathbb{R}^{C \times H \times W}$ are the globally modulated NIR features and the RGB ones, respectively. 

In order to keep simplicity and capture more local information, LMM is built upon GMM by replacing the $1\times1$ convolutional layer in $1\times1$ blocks with a $5\times5$ depth-wise convolutional layer. As its processing is similar to GMM, we do not elaborate on it cumbersomely. 

We will revise the relevant descriptions to make selective fusion clearer in the revision.

Reviewer comment (Presentation):
Table 7 is not mentioned in the paper.

Author response:
`4.` **Table 7 is not mentioned in the paper.**

Thanks for pointing it out, we will rectify it in the revision by illustrating Table 7 in Sec. 6.2.

Reviewer comment (Novelty):
Relative to other NIR-assist denoising approaches, the innovation quotient of the proposed method appears constrained, potentially limiting its adaptability across diverse imaging conditions and noise levels.

Author response:
`1.` **Overall contributions and innovation quotient.**

Previous related works perform NIR-assist image denoising on synthetic noisy images. We kindly remind the reviewer that the most important contribution of this work is to introduce this into the real world. We will revise the paper to make the contribution clear. 

Moreover, other main contributions can be summarized as follows:  

(1) Towards the content inconsistency issue, we propose a simple yet efficient selective fusion module (SFM) that can be seamlessly integrated into existing denoising networks with few increasing computation costs.  

(2) Towards the real-world dataset scarcity issue, we construct a real-world NIR-assisted image denoising (NAID) dataset that covers diverse scenarios and various noise levels, which is expected to serve as a benchmark for future works.  

(3) Extensive experiments demonstrate that our method achieves better results than state-of-the-art ones.  

In addition, we promise the dataset, codes, and pre-trained models will be publicly available, which may be advantageous for future studies.  

## Rebuttal 2 (ID: DugGyuv40t)
`4.` **Adaptability across diverse imaging conditions and noise levels.**

It is worth noting that the proposed method is tailored to address the color and structure discrepancies between NIR-RGB images, which is an inherent issue of NIR-assisted image denoising. In other words, the issue does not disappear as imaging conditions and noise levels change. Thus, our method has the potential to be adapted to different imaging conditions and noise levels.

On the one hand, the proposed NAID dataset covers three different lighting environments, i.e., 2 lux, 5 lux, and 10 lux respectively. For these lighting conditions, our extensive experiments have validated the effectiveness of our method. For lighting conditions beyond the range, it may be more appropriate to collect new data with our data acquisition pipeline and retrain our models on it.

On the other hand, we have conducted extensive experiments on both synthetic DVD [a] and real-world NAID datasets, where the former covers two noise levels and the latter covers three ones. The results show that the proposed method enables performance improvement for all noise levels. And the noise level the higher, the improvement is more significant. It demonstrates that the proposed method can be effectively applied to different noise levels.

Reviewer comment (Experiments):
Concerns arise regarding the efficiency of the methodology, particularly when juxtaposed with alternative techniques, as evidenced by its prolonged training durations.

Author response:
`3.` **Training and inference efficiency.**

We compare the proposed method with the most competitive one DVN [a] as well as the naive feature fusion manner (i.e., feature summation). It is worth noting that we retrain all compared methods with the same configurations, thus the comparisons are fair. As shown in the following table, our method achieves higher performance with lower training time than the compared ones, which illustrates the training efficiency of the proposed method.
| Methods | Training Time | Inference TIme | PSNR / SSIM / LPIPS |
|:---:|:---:|:---:|:---:|
| DVN | 44h 50min | 761ms | 24.93 / 0.7578 / 0.332 |
| Summation | 30h 6min | 461ms | 25.00 / 0.7628 / 0.304 |
| Ours | 30h 13min | 462ms | 25.26 / 0.7676 / 0.303 |

Moreover, we also report the inference time and \#Flops when generating a $1792 \times 1008$ output in the following table. It shows that our method outperforms state-of-the-art ones in inference computation cost, which further illustrates the inference efficiency of our proposed method.
| Methods | #FLOPs | Inference Time |
|:---:|:---:|:---:|
| FGDNet [b] | 38.67G | 476ms |
| SANet [c] | 161.06G | 2761ms |
| CUNet [d] | 14.48G | 542ms |
| MNNet [e] | 23.68G | 1360ms |
| DVN [a] | 104.05G | 761ms |
| Ours | 13.17G | 462ms |

Reviewer comment (Writing):
The introduced SFM module lacks explicit clarity on addressing pivotal challenges inherent to NIR-assist operations.

Author response:
`2.` **The introduced SFM module lacks explicit clarity on addressing pivotal challenges inherent to NIR-assist operations.**

Here we elaborate on the motivations of SFM.  

Color and structure inconsistency issues between NIR-RGB images are the main challenges in NIR-assisted image denoising. First, the NIR image is captured under additional NIR light and is monochromatic, which leads to color discrepancy between NIR-RGB images. Second, the NIR image may 'more-see' or 'less-see' the objects than the visible light ones, primarily due to inherent differences in the optical properties within each spectral domain, which leads to structure discrepancy between NIR-RGB images.  

On the one hand, we hope to design a module to address two inconsistency issues in this work. Thus, we present a global modulation module (GMM) and a local modulation module (LMM) to handle the color and structure inconsistencies, respectively. The combination of the two modules forms SFM. Moreover, the visualization results in Figure A of the appendix show that GMM indeed enables better color and LMM indeed enables better structure.  

On the other hand, we hope the module can be simple, efficient, plug-and-play. Thus, we intentionally avoid some complex operations (e.g., self-attention and cross-attention) during design in order to maintain efficiency as much as possible.  

In addition, in the second response of '**Response to Reviewer YVkk**', we further elaborate on the advantages and characteristics of the proposed SFM.  

We will revise the relevant descriptions to make SFM clearer in the revision.

Reviewer comment (Evaluation):
How does the proposed SFM compare to other denoising techniques in terms of computational efficiency?

Author response:
`3.` **Training and inference efficiency.**

We compare the proposed method with the most competitive one DVN [a] as well as the naive feature fusion manner (i.e., feature summation). It is worth noting that we retrain all compared methods with the same configurations, thus the comparisons are fair. As shown in the following table, our method achieves higher performance with lower training time than the compared ones, which illustrates the training efficiency of the proposed method.
| Methods | Training Time | Inference TIme | PSNR / SSIM / LPIPS |
|:---:|:---:|:---:|:---:|
| DVN | 44h 50min | 761ms | 24.93 / 0.7578 / 0.332 |
| Summation | 30h 6min | 461ms | 25.00 / 0.7628 / 0.304 |
| Ours | 30h 13min | 462ms | 25.26 / 0.7676 / 0.303 |

Moreover, we also report the inference time and \#Flops when generating a $1792 \times 1008$ output in the following table. It shows that our method outperforms state-of-the-art ones in inference computation cost, which further illustrates the inference efficiency of our proposed method.
| Methods | #FLOPs | Inference Time |
|:---:|:---:|:---:|
| FGDNet [b] | 38.67G | 476ms |
| SANet [c] | 161.06G | 2761ms |
| CUNet [d] | 14.48G | 542ms |
| MNNet [e] | 23.68G | 1360ms |
| DVN [a] | 104.05G | 761ms |
| Ours | 13.17G | 462ms |

Reviewer comment (Experiments):
Can the proposed method be applied to other types of images, such as medical images or satellite images?

Author response:
`5.` **Application to other types of images.**

The proposed method is elaborately designed to overcome the color and structure discrepancies between NIR-RGB images. It is simple yet efficient for real-world NIR-assisted image denoising. Simultaneously, we indeed do not know so much about other image types (e.g., medical images and satellite images) and tasks. However, when a similar content inconsistency issue is presented in other images or other tasks, we believe that our method will be also valuable.

Reviewer comment (Experiments):
How does the NAID dataset compare to other benchmark datasets in terms of diversity and size?

Author response:
`6.` **Comparison between NAID and other benchmark datasets in diversity and size.**

As far as we know, for real-world NIR-assisted denoising, Dark Flash Photography [f] is the only publicly available dataset. However, it only consists of five image pairs. It is hard to satisfy data-driven deep learning methods. 

In contrast, the proposed NAID dataset includes 300 NIR-RGB paired images, and covers diverse real-world scenarios and various noise levels, which has the potential to serve as a benchmark for further studies. Moreover, we promise the proposed dataset will be publicly available, which may be advantageous for future studies.

Reviewer comment (Experiments):
There is one place that is not clear. The authors claim that "Therefore, we deploy a GMM to handle color discrepancy first followed by an LMM dealing with structure discrepancy". Is there any supporting evidence that the GMM handles the color discrepancy while the LMM handles the structure discrepancy?

Author response:
We thank the reviewer for the valuable comments and suggestions. We appreciate the reviewer's questions and hope our responses can address the concerns. 

`1.` **Evidence to support that GMM handles the color discrepancy while LMM handles the structure one.**

We have provided the visualization comparison results in Figure A of the appendix by removing GMM and LMM separately. It shows that GMM results in better color recovery and LMM results in better structure recovery, which supports that GMM handles the color discrepancy and the LMM handles the structure one.

Reviewer comment (Experiments):
The ablation experiment lacked a probe to see if the loss of multiple scales would affect the final output picture.

Author response:
`1.` **Effect of multi-scale loss.**

We apologize for the lack of explanations and effect of multi-scale loss. The multi-scale loss calculates the difference between output and supervision at different scales, and it can slightly improve performance with few increasing training costs. Taking NAFNet [a] as an example, we evaluate its effect by comparing it with naive $\ell_2$ loss. The following table shows that it improves both single-image and NIR-assisted denoising performance. And the improvement is more obvious for the latter. It is worth noting that we conduct all ablation studies with the multi-scale loss, thus the comparisons between them are fair. 
| Denoising Methods | Input Images | Loss Function | PSNR / SSIM / LPIPS |
|:---:|:---:|:---:|:---:|
| Single-Image | RGB | Naive $\ell_2$ | 24.72 / 0.7476 / 0.343 |
| Singel-Image | RGB | Multi-Scale  | 24.73 / 0.7483 / 0.336 |
| NIR-Assisted | RGB and NIR | Naive $\ell_2$ | 25.20 / 0.7691 / 0.299 |
| NIR-Assisted | RGB and NIR | Multi-Scale | 25.26 / 0.7695 / 0.299 |

We will add this in the revision.

Reviewer comment (Novelty):
The lightweight fusion module in this paper does not contribute enough to the technology and simply fuses the features.

Author response:
`2.` **The lightweight fusion module does not contribute enough and simply fuses the features.**

Here we elaborate on the proposed selective fusion module (SFM) to further show its contributions. 

(1) The proposed SFM is a problem-oriented design module.

Color and structure discrepancies between RGB and NIR images are the main challenges of NIR-assisted denoising. SFM is elaborately designed towards the issue. Specifically, it consists of a global modulation module (GMM), a local modulation module (LMM) and a fusion operation, where GMM modulates features globally to overcome the color discrepancy and LMM does that locally to overcome the structure one. 

(2) The proposed SFM is a simple yet efficient module.

Extensive quantitative and qualitative experiment results show that the proposed method outperforms state-of-the-art ones. Besides, we have tried some prevailing methods like feature summation, channel attention [b],  self-attention [c], and cross-attention [c] in experiments. However, as shown in the following table, they do not achieve competitive performance while introducing higher computation costs.
| Fusion Methods | PSNR / SSIM / LPIPS | Inference Time |
|:---:|:---:|:---:|
| Summatioin | 25.03 / 0.7647 / 0.304 | 461ms |
| Channel Attention | 25.12 / 0.7664 / 0.309 | 461ms |
| Self-Attentnion | 25.12 / 0.7659 / 0.307 | 1614ms |
| Cross-Attention | 25.18 / 0.7692 / 0.304 | 1712ms |
| Ours | 25.26 / 0.7695 / 0.299 | 462ms |

(3) The proposed SFM is a plug-and-play module. 

It can be seamlessly integrated into existing denoising networks with few increasing computation costs.

Combining the above reasons, we finally choose the simple yet effective, plug-and-play SFM as our solution. Besides, we propose a real-world NIR-assisted image denoising (NAID) dataset that covers diverse scenarios as well as various noise levels and is expected to serve as a benchmark for future studies. More importantly, we introduce NIR-assisted image denoising from a synthetic manner into the real world.

We will revise the relevant descriptions to make SFM clearer in the revision.

Reviewer comment (Experiments):
This paper does not demonstrate whether the improvement in denoising performance is due to the superiority of the method or only to the inclusion of NIR information.

Author response:
`3.` **Is the performance improvement due to the method's superiority or only to the inclusion of NIR information?**

Here we provide a detailed and clearer description of that, and we will add it in the revision.  

We take NAFNet as a denoising backbone and compare our method with a single-image denoising manner (only using RGB images) and a naive fusion manner (i.e., simple summation of RGB features and NIR ones). The results on NAID dataset are shown in the following table. We employ PSNR, SSIM and LPIPS as evaluation metrics. 

On the one hand, the inclusion of NIR information improves RGB image denoising, even simply fusing their features by summation. On the other hand, simple summation is not the optimal solution as it ignores color and structure discrepancies between NIR-RGB images. Taking this into account, the proposed SFM further improves performance, and the improvement is more significant when the noise level is higher. It illustrates the superiority of the proposed method.
| Input Images | Fusion Methods |  Low-Level Noise | Middle-Level Noise | High-Level Noise |
|:---:|:---:|:---:|:---:|:---:|
| RGB | None |  25.71 / 0.7780 / 0.294 | 24.76 / 0.7482 / 0.335 | 23.71 / 0.7186 / 0.378 |
| RGB and NIR | Summation |  25.92 / 0.7883 / 0.277 | 25.00 / 0.7682 / 0.304 | 24.16 / 0.7429 / 0.329 |
| RGB and NIR | Our SFM | 26.06 / 0.7906 / 0.274 | 25.26 / 0.7676 / 0.303 | 24.48 / 0.7503 / 0.321 |

Reviewer comment (Experiments):
Are the methods compared in the paper all image restoration assisted by NIR images? If not, is retraining performed for the comparison?

Author response:
`4.` **Whether all compared methods are assisted by NIR images? Whether they are retrained for the comparison?**

In Tables 2 and 3, the methods marked as 'Single-Image Denoising' in the top three rows do not use NIR images, and the bottom eight ones marked as 'NIR-Assisted Denoising' remove noise assisted by NIR images. 

All compared methods are retrained with the same configurations on synthetic DVD [d] and our real-world NAID datasets respectively. Thus, the comparisons are fair.

Compared to the most competitive one DVN [d],  our method achieves 0.39dB PSNR gain when $\sigma = 4$ and 0.43dB PSNR gain when $\sigma = 8$ on DVD dataset. It also achieves 0.1dB, 0.33dB, and 0.53dB improvement on PSNR in dealing with low-level, middle-level, and high-level noise on the NAID dataset, respectively.