Paper: UniINR: Unifying Spatial-Temporal INR for RS Video Correction, Deblur, and Interpolation with an Event Camera (ICLR-2024)

Reviewer comment (Theory):
The fusion of events and images is completely end-to-end, with no explicit connection between the exposure time and the timestamp in events. This looks like a whole black box and lacks interpretability. There are established methods that associate exposure time, optical flow, and events, which is more intuitive, but of course introduces more computation, which is a key claim of this method.

Author response:
### W1.1: NO explicit connection between the exposure time and the timestamp in events.
We apologize for any confusion caused by our presentation.
To clarify, we do indeed model the relationship between exposure time and the timestamp of events in our paper (Section 3.1, Lines 133-169).
Specifically, Equations 3 and 4 in our paper introduce the connection between exposure time and timestamps. Additionally, the role of the event in this formulation is detailed from Line 158 to Line 160.
This approach is the linkage between the eve timestamp of events and the image exposure time.

### W1.2: Associate exposure time, optical flow, and events, which is more intuitive.
Furthermore, our INR-based method utilizes Exposure Time Embedding to incorporate time information into the INR. This allows us to directly obtain intensity values at any given time, thereby circumventing the need for more complex operations like optical flow computation. As you pointed out, while such methods are more intuitive, they also introduce significantly more computational complexity. Our approach achieves comparable results with fewer parameters, aligning with our goal of computational efficiency.

Reviewer comment (Evaluation):
Visual comparisons on real dataset does not show significant differences. Are there more obvious samples in the dataset? Otherwise I don't think it can support the generalizability of the model.

Author response:
### W2: More obvious samples in the dataset.
Thank you for pointing out the need for more distinct examples in our dataset. We acknowledge your concerns regarding the visual comparisons on the real dataset. To address this, we direct your attention to Figure 4 (c) in our paper, where the results from EvUnroll exhibit noticeable artifacts, as indicated by the red circles. These artifacts underscore the differences and improvements our method offers.

Furthermore, we have included additional real-world data visualizations in the supplementary materials (Figure 6). These examples effectively demonstrate our method's ability to correct deformations, particularly in challenging scenarios such as outdoor windows. This robust performance in diverse real-world conditions highlights the generalizability and effectiveness of our rolling shutter correction approach.

Reviewer comment (Experiments):
In C.5 of the Supplementary Material, the author mentions that concat can perform better than addition, so why does the author continue to use only addition?

Author response:
### W3: Concat and addition operate:
Thank you for your insightful query regarding our choice of operation in our model.
Indeed, while concatenation can offer marginal improvements, it comes at the cost of increased model size and computational overhead. In our experiments, we observed that the performance differences between addition, multiplication, and concatenation were relatively minimal, with variations around 0.03 dB and 0.04 dB, which are not substantial enough to justify the additional complexity.

Given our emphasis on maintaining a lightweight and efficient design, we decided to continue using addition, which aligns with our objectives of computational efficiency and simplicity. However, recognizing the potential benefits of alternative operations, we plan to include these options in our publicly released code. This will allow users to explore and leverage different operation types according to their specific requirements and constraints.

Reviewer comment (Novelty):
Although the question raised is forward-looking, the technical details given by the authors are not very innovative, like some stacking of existing models. This makes it harder for reviewers to capture what the real contributions are. It is recommended that authors summarize the contribution visibly.

Author response:
We are grateful for your feedback and take this opportunity to succinctly restate the key contributions of our work, earnestly anticipating your recognition of these aspects:

**Novel Research Question:** This study pioneers the recovery of high-frame-rate sharp GS frames at arbitrary frame rates from RS blur images and paired event data. Our approach diverges fundamentally from traditional methods, charting a new course in event-based imaging. This first-of-its-kind effort brings a fresh perspective to the field.

**Innovative Methodology:** We have developed a one-stage framework utilizing a unified implicit neural representation (INR). Distinctively, it maps space-time coordinates to RGB values through an Exposure Time Embedding module coupled with a Pixel-by-pixel Decoding module. This framework eschews the conventional reliance on optical flow, traditionally known for its computational complexity. Our innovative integration of temporal and spatial data sets our method apart from existing approaches.

**Superior Performance:** Our model demonstrates exceptional performance across datasets with varying degrees of blur, as evidenced in Tables 1 and 2 of the main paper. It consistently outperforms the existing method, EvUNRoll, while maintaining a remarkably efficient architecture with only 0.379M parameters. This efficiency, combined with its effectiveness, underscores our model's superiority over more complex models.

Reviewer comment (Experiments):
A more important issue is that of experimental validation. Overall experimental validation is not sufficient to prove the validity of the proposed methodology. The authors mention the use of two different experimental settings: (I) the single GS sharp frame restoration, i.e., the RS correction and deblurring. However, the EvUnroll is only the event-guided RS correction method, and eSL-Net is only the event-guided deblurring method. The target algorithm that the authors should be comparing should be correction+deblurring simultaneously. Thus, it is recommended that the authors use a combination of correction + deblurring algorithms for experimental comparisons, including but not limited to EvUnroll + eSL-Net; there are many event-guided deblurring algorithms available. (II) a sequence of GS sharp frames restoration, i.e., the RS correction, deblurring and interpolation. However, the DeblurSR is the event-guided deblurring and interpolation method and does not target the correction. Therefore, the experimental comparison is not very fair. Meanwhile, the combination of EvUnroll and TimeLens does not target the deblurring. Overall, the authors are advised to conduct a fairer and more comprehensive experimental comparison.

Author response:
We value your thoughtful feedback regarding our experimental validation. To clarify and address your concerns:

**Single GS Sharp Frame Restoration:**

- **EvUnroll:** As stated in EvUnroll's documentation, it includes an optional image deblurring module. By processing the RS image first through this deblurring module and then the RS correction module, EvUnroll effectively accomplishes both deblurring and RS correction in sequence.
- **eSL-Net:** For our specific experimental needs, we have adapted eSL-Net by modifying its parameter initialization method and removing the up-sampling module. These adjustments ensure its compatibility and efficient functioning in our experimental framework.

**Sequence of GS Sharp Frames Restoration:**
- **Combination of EvUnroll and TimeLens:** This combination is capable of handling deblurring, RS correction, and interpolation. The process involves sequentially feeding inputs through EvUnroll’s optional deblurring module, then its RS correction module, and finally, TimeLens’s interpolation network. For a detailed depiction of this procedure, we refer you to **Fig. 10 (II)** in the supplementary material, which offers a comprehensive illustration of the entire process.

We trust these detailed explanations and methodological adjustments adequately address your concerns regarding the fairness and thoroughness of our experimental comparisons.

Reviewer comment (Experiments):
Whether the algorithms being compared are using pre-trained models or re-trained? Reviewers noted that the compared methods were significantly lower than the proposed methods.

Author response:
Response: We appreciate your insightful observations regarding the training status of models in our comparative analysis. We recognize the importance of clear experimental descriptions and are committed to enhancing the clarity in the experimental section of our paper. To address your concerns, we offer the following clarifications:

**Pre-trained Models or Re-trained?:**
We utilized the pre-trained model of TimeLens owing to the unavailability of its training code.
For all other models in our comparison, we re-trained them using their respective publicly released codes.

**Fairness in Comparative Analysis:**
To ensure an equitable comparison, we evaluated our method under the same settings as those used for EvUnroll.
This comparison, detailed in Table 2 of our main paper, underscores the superior performance of our approach.
It’s noteworthy that EvUnroll supports the simulated dataset used in Table 2.

**Focus on larger degrees of Motion Blur and Distortion:**
Benefiting from the one-stage method, our model has better performance for large degree motion blur.
To this end, we created a simulated dataset with a larger degree of blur, as presented in Table 1.
The parameters for this simulation are detailed in Supplementary Material C.1.
As indicated in Table 1, the performance of the compared methods falls below that of our proposed method.
This discrepancy can be attributed to the inherent challenges associated with effectively handling severe RS correction, motion deblurring, and frame interpolation simultaneously.

Reviewer comment (Novelty):
In the fusion of RS frames and event streams, there is no specific modules or design that focus on the domain gap between these two modalities. Since the format and information recorded by the intensity values and event signals are quite different.

Author response:
In response to your query about the fusion of RS frames and event streams, we would like to direct your attention to lines 181-198 of our paper, where we elaborate on the rationale behind our encoder's design. Our approach is influenced by the principles of sparse learning, which we've found to be particularly effective when bridging the domain gap between the two modalities. The sparse learning framework has been proven effective in previous work for handling the distinct formats and information characteristics inherent in intensity values and event signals. By integrating this framework into our encoder design, we effectively address the unique challenges posed by the fusion of these two distinct data types.

Reviewer comment (Theory):
The event signals directly record the coordinates of events triggered at that time. Unlike the intensity frames, in one event voxel tensor, not all the pixels contain event signals. As a result, directly applying INR to event voxel tensor doesn’t consider the sparsity of event signals.

Author response:
In addressing your concern regarding the application of INR to event voxel tensors, it's important to clarify that our INR framework operates on the features outputted by the Encoder, rather than directly on the event data. The Encoder is specifically designed to capitalize on the high temporal resolution of event signals, enabling it to learn a continuous spatiotemporal representation. This design choice effectively addresses the sparsity of event signals, ensuring that our INR framework is applied to a more contextually rich and temporally informed feature set, rather than directly to the sparse event data.

Reviewer comment (Novelty):
The authors claim for multiple times that they are the first to achieve arbitrary frame interpolation from an RS blur image and events. I think it is not that convincing. Because EvUnroll is also able to reconstruct multiple GS sharp frames.

Author response:
We appreciate the opportunity to clarify this aspect of our work. While EvUnroll is indeed capable of interpolating frames, its capability is not as flexible as our method in terms of generating frames at arbitrary timestamps. Our approach uniquely enables the interpolation of frames at any specified time point, which is a distinct advancement over EvUnroll's capabilities. This key difference is demonstrated in Figure 1 of our main paper and further elaborated in Figures 9 and 10 of the supplementary material. Our method’s ability to interpolate frames at any given time, irrespective of the original frame rate or timing, sets it apart from EvUnroll and underscores the novelty of our contribution in this area.

Reviewer comment (Reproducibility):
In the supplementary code, I found that in the SCN module, the input images and events are multiplied to get a feature called ‘x1’. I wonder what the meaning of multiplication is.

Author response:
Thank you for your inquiry regarding the multiplication operation in the SCN module within our supplementary code. This multiplication between input images and events to obtain the feature 'x1' is a key aspect of our sparse learning approach. This method is inspired by and closely references the techniques outlined in the previous paper on eSL (Event Sparse Learning). By multiplying the intensity frames with the event data, we effectively integrate the high temporal resolution information from the event signals with the spatial information from the intensity frames.
This integration is crucial for capturing the dynamic changes in the scene, a cornerstone of our method's ability to accurately interpret and process the event data in conjunction with traditional intensity frames.

Reviewer comment (Experiments):
In experimental part, this paper does not utilize optical flow for frame interpolation, unlike methods such as VideoINR [1] and MoTIF [2], which use INR to predict optical flow and then perform frame interpolation. However, in this paper, optical flow is not employed, and it seems that there is no dedicated comparison with optical flow methods in the conducted ablation experiments.

Author response:
Thank you for highlighting the differences in approach between our work and methods like VideoINR and MoTIF. We would like to clarify our methodology and its distinctions:

**Different Research Focus:**
VideoINR and MoTIF are primarily focused on super-resolution (SR) and video frame interpolation (VFI), where warp and multi-frame fusion are applicable and effective. Our research, however, concentrates on addressing the challenges of blur and rolling shutter (RS) distortion. Due to the nature of these distortions, warp and multi-frame fusion, as used in VideoINR and MoTIF, are theoretically not feasible for our specific problem setting.

**Unique Methodological Approach:**
The research problems we tackle necessitate fundamentally different methodologies compared to VideoINR. While VideoINR integrates SpatialINR and TemporalINR sequentially for frame interpolation and super-resolution, our method innovates by developing a unified INR strategy. This unified INR incorporates Exposure Time Embedding and Pixel-by-Pixel Decoding, enabling simultaneous RS correction, deblurring, and frame interpolation. This approach is specifically tailored to address the complexities of RS deformation and blur, an aspect not considered in VideoINR.
Our Exposure Time Embedding module is a crucial differentiator, allowing for the generation of both RS and GS images with specific exposure time information. This capability is unique to our approach and is not achievable by VideoINR, given their different focus.

**Different Experimental Dataset:**
Owing to the divergent research objectives and methodologies, the dataset used in our study is distinct from that utilized in VideoINR. The choice of dataset in our research is aligned with our goals to specifically address RS correction, deblurring, and frame interpolation under conditions of RS distortion and blur.
By elaborating these key differences, we hope to provide a clearer understanding of our method's unique contributions and the rationale behind our experimental choices.

Reviewer comment (Experiments):
A comprehensive performance comparison with EvShutter[1] is suggested, which has been published in CVPR2023.

Author response:
We appreciate your suggestion regarding a comprehensive performance comparison with EvShutter. We would like to provide the following clarifications to address this concern:

**Accessibility of the Dataset:** The RS-ERGB dataset proposed by EvShutter is not publicly available, limiting our ability to perform a direct comparison.

**Opacity of Dataset Details:** The EvShutter paper does provide comparative results on the Fastec-RS dataset. However, essential details about this dataset, specifically the degree of simulation for Rolling Shutter and Blur, are not explicitly disclosed. This ambiguity impedes our ability to perform a precise and accurate comparative analysis with EvShutter.

**Lack of Code and Implementation Details:** Another challenge is the unavailability of EvShutter's code. Additionally, the paper omits certain critical implementation details. These gaps pose a considerable obstacle in ensuring an accurate and equitable comparison.

Despite these hurdles, we have endeavored to present a general comparison. In the Supplementary Material C.1 (lines 539-549), we describe our approach for synthesizing Rolling Shutter and Blur on the Fastec-RS dataset. Our experimental results demonstrate robust performance even under conditions of more pronounced blurring than what is seen with EvShutter (and EvUnRoll). Specifically, our model achieves a PSNR value of 33.64 and an SSIM value of 0.9299. These metrics significantly surpass the performance of 32.41 PSNR and 0.91 SSIM reported by EvShutter, highlighting the efficacy of our method under challenging conditions.

Reviewer comment (Experiments):
Both EvShutter and Evunroll demonstrate the performance of the proposed algorithms for eliminating the RS effect caused by fan rotation. To verify the robustness of the proposed method for different scenes, it is suggested to add similar experiments on real fan data.

Author response:
Thank you for suggesting additional experiments with real fan data to demonstrate the robustness of our method in different scenes. We acknowledge that EvUnRoll reported fan samples in their paper. However, the four real-world data scenes provided by EvUnRoll do not include these fan examples. In our experiments, we focused on these four scenarios, two indoor and two outdoor, to evaluate our method's effectiveness in diverse environments. The results of these experiments are showcased in Figure 6 of our supplementary materials. Our model demonstrates effective correction in both indoor and outdoor settings, with noticeable corrections observed especially in outdoor window scenes. This performance highlights our model's capability to adapt and correct various real-world distortions, reinforcing its robustness and versatility.

Reviewer comment (Experiments):
In Figure 5, it seems unfair and unnecessary to compare the running time of the proposed algorithm with the combination of EU and TL.

Author response:
We appreciate your observation regarding the comparison made in Figure 5. We understand the concern about the fairness and necessity of comparing our algorithm's running time with the combination of TL (TimeLens) and EvUnRoll. It's important to note that our method represents the first end-to-end solution for RS deblurring and frame interpolation (VFI), which presents a challenge in finding directly comparable methods.

Given this context, we selected TL + EvUnRoll as the best available comparison, despite the differences in approach. This choice was made to provide a benchmark, albeit not a perfect one, against established methods in the field.

To ensure the fairness of the speed test, as detailed in lines 334 to 342 of the main paper, we carefully avoided the inclusion of I/O times and focused solely on the running time on the GPU. This approach was taken to provide as fair and accurate a comparison as possible, given the constraints of available methods for comparison. Our goal was to offer a meaningful context for the efficiency of our end-to-end solution, even though a perfectly analogous method does not currently exist.