Paper: Self-Supervised High Dynamic Range Imaging with Multi-Exposure Images in Dynamic Scenes (ICLR-2024)

Reviewer comment (Theory):
The proposed method is specific to 3 inputs. The 3 inputs should be informatively captured, especially the mid-exposed image, which should have texture and color of high fidelity as the reference. The requirement for the input is strict and can limit the application of the proposed method.

Author response:
We thank the reviewer for the valuable comments and suggestions.
We appreciate the reviewer's questions and hope our responses could address the concerns.


1. Limitation of Inputs

We agree with the reviewer's insight on method limitations. Moreover, most existing multi-exposure HDR reconstruction methods (including supervised and self-supervised ones) only focus on removing ghosting artifacts caused by misalignment between inputs, having this limitation as well. This work (SelfHDR) has taken a step toward more realistic self-supervised HDR imaging by deghosting, while our ongoing work is to further address this limitation.


Here we share two possible solutions. On the one hand, we can combine HDR reconstruction with some self-supervised image restoration methods when considering more realistic shooting conditions. For example, there may exist noise in short-exposure images and blur in long-exposure images. In order to achieve a self-supervised algorithm, we can combine HDR reconstruction with some self-supervised denoising and debluring works to process input images for removing degradations. 

On the other hand, an adaptive method may need to be explored to select a more appropriate image as a base frame. For example, when a mid-exposure image suffers more severe degradations than others, the method should adaptively take short-exposure or long-exposure images as a new base frame for HDR reconstruction. Thus, it can alleviate the shooting requirements for a mid-exposure image.

We will add this in Sec. E of the appendix in the revision.

Reviewer comment (Experiments):
The structure-focused network was proposed to avoid errors caused by optical flow alignment, but it is integrated with optical flow alignment in training stage 2, which doesn’t make sense to me. Although Tab. 6 shows improvement in general, I think it will hurt the performance in the cases when the alignment has more errors, whereas those cases can demonstrate the core value of the paper. The authors should elaborate on this and experiment on those cases to justify their statement when necessary.

Author response:
We thank the reviewer for the valuable comments and suggestions.
We appreciate the reviewer's questions and hope our responses could address the concerns.


2. Pre-Alignment in Structure-Focused Network



The ultimate goal of the structure-focused network is to provide HDR supervision with good structure that is complementary to the color component. It should avoid ghosting results, rather than avoid errors caused by optical flow alignment. Moreover, different from naive merging operation (in Eqn. (5)) for generating color component, learning-based structure-focused network has the ability to handle misalignment problem, including the remaining misalignment after optical flow pre-alignment, as shown in Fig. A (l).



Below we further explain the effect of pre-alignment. On the one hand, significantly incorrect optical flow mainly appears in occlusion and overexposed areas from other frames. These areas have little valuable information, so the results mainly rely on the base frame, and pre-alignment or not has little impact on the results. Thus, it's not bad to perform pre-alignment in this case. On the other hand, when optical flow is roughly correct, pre-alignment reduces the processing burden of the structure-focused network that does not adopt explicit alignment. Thus, it naturally brings improvements.



We additionally test the structure-focused network on 74 training scenes with and without optical flow pre-alignment, respectively. First, the average results are shown in the following table. It can be seen that the pre-alignment manner has a significant improvement on average. Second, we compare the results between the two manners one by one. We find that only in 6 scenes, the results without pre-alignment are more than 0.1dB better than those with pre-alignment on PSNR-$u$. In the other 68 scenes, the pre-alignment manner always gives better or comparable results. Moreover, we also observe the characteristics of these 6 scenes. We find that their alignment is not worse than the other 68 scenes and is moderate in general. The experiments show that the above explanation is reasonable in most scenes.



|  | PSNR-$u$ / SSIM-$u$ | PSNR-$l$ / SSIM-$l$ 
| :----:       |  :----:  | :----:  |
| w/o Pre-Alignment        | 42.40 / 0.9809    | 40.02 / 0.9869   |
| w/ Pre-Alignment       | 42.84 / 0.9814    | 41.48 / 0.9894   |

We will add this in Sec. B of the appendix in the revision.

Reviewer comment (Presentation):
In Tab 3-6 the highest scores can be marked in bold.

Author response:
We thank the reviewer for the valuable comments and suggestions.
We appreciate the reviewer's questions and hope our responses could address the concerns.


3. Paper Writing and Misc

Many thanks. In the revision, we will add pointers to the contents of the appendix. We will also briefly introduce HDR-VDP-2 and mark the best results in Tables 3$\sim$6 in bold.

Reviewer comment (Evaluation):
The paper could have also delved more into the limitations of the proposed method, such as the cases where the SelfHDR method may not provide optimal results.

Author response:
In addition to handling noisy images, here we give some other situations where SelfHDR may fail. First, when long-exposure images suffer from blur, SelfHDR may introduce blurry results. In future work, we can combine some self-supervised image restoration methods to process the blurry images. Second, when the scene irradiance changes drastically in shooting multi-exposure images, SelfHDR may fail, as the constructed color components may be inaccurate. And this situation is also difficult to handle for supervised methods.

Reviewer comment (Experiments):
An exploration of the generalizability limitations of the method would have been appreciated.

Author response:
Besides, SelfHDR has no specific requirements for scene type and content color. In terms of generalization ability in different scenes, SelfHDR is similar to the corresponding supervised HDR reconstruction networks. And the best way to improve generalization ability is to collect more data and retrain the network.

We will add this and give some examples in Sec. E of the appendix in the revision.

Reviewer comment (Experiments):
How sensitive is the SelfHDR method to the quality of the input images (for example, noise)? Would substantial noise or minor shifts in alignment between images drastically affect the performance of the method? This might be a bit hard to comment on, as the misalignments in the Kalantari dataset are already fixed.

Author response:
1. SelfHDR Sensitivity to Noise

SelfHDR does not specifically deal with the noise problem. If the noise in the multi-exposure image is obvious, the result may also contain noise. Moreover, most existing multi-exposure HDR reconstruction methods (including supervised and self-supervised ones) only focus on removing ghosting artifacts caused by misalignment between inputs, having this limitation as well. SelfHDR has taken a step toward more realistic self-supervised HDR imaging by deghosting, while our ongoing work is to further address this limitation.

Nonetheless, perhaps benefiting from the combination of two complementary supervision, our method still performs better than other self-supervised methods (e.g., FSHDR) when testing noisy images, as shown in Fig. 4(b).


2. SelfHDR Sensitivity to Misalignment


The robustness to misalignment depends on the HDR reconstruction networks we use. In Table 1, we use four HDR reconstruction networks (including CNN-based ones and Transformer-based ones) for experiments. From the results, the Transformer-based networks (i.e., HDR-Transformer and SCTNet) perform better. And the qualitative results testing on out-of-domain datasets (i.e., Sen et al. and Tursun et al. datasets) also show this trend.

In addition, benefiting from the self-supervised properties of SelfHDR, the best way to improve robustness may be to collect as much data as possible for training.

Reviewer comment (Evaluation):
Is the performance upper limit of this work limited by the upper limit of structure or color supervision? What designs correspond to solving this problem?

Author response:
The performance upper limit of our work is indeed limited by the upper limit of structure or color supervision. Actually, the performance of existing supervised methods is also limited by the quality of ground truth (GT). Moreover, although the individual structure or color component is not sufficient as supervision, combining the two components as supervision achieves comparable results to supervised methods. As a self-supervised method, the performance has been enough good.

Below we describe in detail the combination of the two complementary components. Two strategies are presented to combine their advantages and discard their disadvantages. First, we use different loss terms for the two supervisions to learn their different information. We adopt L1 loss for the color component and VGG loss for the structure component. The purpose of the former is to learn color information, while the latter is to learn texture information. Second, we design masks to exclude ghosting areas in color components, thus avoiding adverse impacts from these areas.

Reviewer comment (Theory):
This method may also be limited by the alignment method.

Author response:
The alignment approach mainly influences the color component. Nevertheless, the color component is not very demanding for the alignment approach, as its poorly aligned areas can be further supplemented by the structure component. Furthermore, the development of the current image alignment works is mature, they provide us with enough assistance to achieve comparable performance to supervised methods.

We will add this in Sec. E of the appendix in the revision.

Reviewer comment (Presentation):
Some visualizations of structure and color supervision can be given.

Author response:
We respectively visualize a color and structure supervision in Fig. A (g) and (l) of the appendix.

Reviewer comment (Novelty):
For dynamic scenes, the method relies on existing registration methods to obtain more accurate color components for registration.

Author response:
The color component indeed relies on the registration method. Nevertheless, it is not very demanding for the registration method, as its poorly aligned areas can be further supplemented by the structure component. Furthermore, the development of the current image registration works is mature, they provide us with enough assistance to achieve comparable performance to supervised methods.

We will add this in Sec. E of the appendix in the revision.

Reviewer comment (Experiments):
One of the contributions is the construction of a color component. However, it lacks the comparison of the overall results which can directly reflect this advantage in terms of color.

Author response:
The color component is generated by weighting the aligned images, as shown in Eqn. (5). When some areas are aligned well, the corresponding areas in the color component are consistent with the ground-truth. When some areas are aligned poorly, the corresponding areas in the color component may have ghosting artifacts. For the former situation, we can directly take these well aligned areas as supervision of HDR reconstruction network. For the latter situation, we construct a mask in Eqn. (14) to exclude these poorly aligned areas when training HDR reconstruction network. And experiments in Table 4 show that the mask indeed plays a positive role.

In addition, we visualize a color component in Fig. A (g) of the appendix. It can be seen that its characteristics are consistent with the phenomenon we describe above.

Reviewer comment (Theory):
Does this decomposition way (decomposing an image into color and structure components) have a corresponding theoretical basis, and does it correspond to some existing image decomposition theory?

Author response:
Our decomposition way is a focus or emphasis on color and structure relatively, not an absolute separation. Nevertheless, it may have some similarities with Laplace decomposition. The color component is similar to the low-frequency part of  Laplace decomposition, while the structure component is similar to the high-frequency part of Laplace decomposition.

Reviewer comment (Theory):
The mask-based weighting does not consider pixel neighborhood relationships. Will it lead to artifacts?

Author response:
Fig. 1 is a commonly used weighting function for generating an HDR image from static multi-exposure images. In Fig. 1, the abscissa of the function figure is the medium-exposure image value, and the ordinate is the mask value. In other words, the mask value depends on the image value. As long as the images are aligned well, additional artifacts will not be introduced in the color component. Moreover, we visualize a mask in Fig. A (i) of the appendix. It can be seen the mask is highly relevant to the image value, and is not scattered and irregular.