Paper: Exploring the Common Appearance-Boundary Adaptation for Nighttime Optical Flow (ICLR-2024)

Reviewer comment (Evaluation):
It would be better to add some analyses on the failure cases and show the limitations of the method and give some discussions.

Author response:
**Q1:** Limitation of the proposed method.

**A:** Thanks for appreciating our work. We have specifically discussed the limitation of the proposed method in the "3.7 Limitation'' subsection of the supplementary material. The proposed method can model the x, y-axis motion pattern, but fails to estimate the radial motion along the z-axis. The key to z-axis radial motion estimation is to obtain the accurate scene depth information. From the perspective of imaging mechanism, the frame camera records absolute luminance in the x, y-axis via global scan, while the event camera mainly triggers x, y-axis events due to brightness change, and is difficult to respond to z-axis motion in a short time. Therefore, it is difficult for frame and event cameras to directly perceive z-axis motion information. For example, when imaging that a vehicle is approaching us from far to near along the center of the camera, the frame camera cannot capture the relative motion of the vehicle, and the event camera also fails to trigger the corresponding events. From the perspective of knowledge transfer, the common space we constructed follows the x, y-axis basic optical flow model, serving as an intermediate bridge to guide the knowledge transfer, which lacks the constraint on z-axis motion knowledge. Both aspects limit the ability of the proposed method to model z-axis radial motion. In the future, we will further introduce LiDAR which is robust to illumination, to directly measure the depth information to assistantly estimate the 3D motion.

Reviewer comment (Experiments):
Can the proposed method be applied to other challenging optical flow estimation tasks, such as low-light or high-speed scenarios?

Author response:
**Q3:** Application for low-light and high-speed scenarios.

**A:** Low light and high speed are two of the more challenging scenarios for nighttime scenes. In terms of imaging mechanism, conventional frame camera records the absolute luminance with a fixed exposure time via global scan. In nighttime scenes, conventional frame camera would inevitably face a dilemma between the long exposure time of low-light scenarios and motion blur of high-speed scenarios. In contrast, the event camera reacts to changes in light intensity, rather than integrating photons during the exposure time of each frame [2]. Each pixel works independently and returns a signal only when an intensity change is detected. Compared with the conventional frame camera, the event camera can sense the dynamic changes with higher temporal resolution (microsecond) and higher dynamic range, thus compensating for frame camera in nighttime scenes, such as, nighttime image enhancement [3] and nighttime deblurring [4].

Similarly, the event camera can also assist the frame camera to learn the motion patterns in nighttime low-light and high-speed scenarios. To discuss the impact of the event on the optical flow in nighttime low-light and high-speed scenes, as shown in the following table, we use the coaxial optical system (seeing Fig. 1 in the supplementary PDF) to collect the spatiotemporally-aligned image sequences and events stream with various illumination (e.g., 3.5 lux, 9.2 lux and 12.7 lux) and various driving speed (e.g., 50 km/h, 70 km/h and 80 km/h) to quantitatively compare the optical flow performance. As for the optical flow label, since it is difficult to directly obtain the dense optical flow labels, we manually mark 100 pairs of corresponding corner points for each two adjacent images, and calculate the relative displacement between the corner points as the sparse optical flow labels. In addition, we choose EPE as the evaluation metric.

|                                              | Low light |         |          | High speed |         |         |
| :------------------------------------------: | :-------: | :-----: | :------: | :--------: | :-----: | :-----: |
|                    Method                    |  3.5 lux  | 9.2 lux | 12.7 lux |  50 km/h   | 70 km/h | 80 km/h |
|              our flow baseline               |   4.26    |  3.98   |   3.67   |    3.65    |  4.53   |  5.74   |
|       our flow baseline, w/ only event       |   2.05    |  2.01   |   1.63   |    1.75    |  1.75   |  1.80   |
| our flow baseline, w/ event, w/ common space |   1.52    |  1.44   |   1.10   |    1.14    |  1.15   |  1.19   |

We have two observations. First, the event camera can greatly improve optical flow performance in both nighttime low-light and nighttime high-speed scenes. In high-speed scenes, the proposed method is robust to various speeds, and the optical flow performance remains unchanged, demonstrating the advantage of high temporal resolution of the event camera. In low-light scenes, as the illumination becomes lower, the optical flow metric (EPE) trend becomes larger obviously. This shows that, although the event camera has the advantage of high dynamic range, too low illumination would also interfere with the optical flow performance. The main reason is that, under low light conditions, the event noise is intensified. The event noise and the valid signal event are both 0-1 pulses, and their difference is very small, which affects the optical flow. In the future, we will further consider the impact of noise and achieve optical flow estimation under extremely low-light scenes. Second, the proposed common space can further improve the upper limit of optical flow, indicating that common space can serve as a bridge to reinforce the feature alignment between event and nighttime image domains.

[2] Cabriel C, et al. Event-based vision sensor for fast and dense single-molecule localization microscopy. Nature Photonics, 2023.

[3] Liang J, et al. Coherent Event Guided Low-Light Video Enhancement. ICCV, 2023.

[4] Qi Y, et al. E2NeRF: Event Enhanced Neural Radiance Fields from Blurry Images. ICCV, 2023.

Reviewer comment (Evaluation):
How does the proposed method handle the trade-off between model size and computational efficiency? Are there any plans to further optimize the runtime or explore other efficient optical flow estimation techniques?

Author response:
**Q2:** The impact of common space adaptation on the model size and efficiency of the optical flow model for inference.

**A:** It should be emphasized that the focus of this work is not the single optical flow backbone in the inference stage, but the knowledge transfer framework in the training stage for the nighttime optical flow task. The final optical flow model only needs the nighttime optical flow encoder $E_n$ and $GRU$ in Fig. 2 of this paper for testing, which can be replaced with any optical flow backbone. The model size and runtime are related to the optical flow backbone, while the accuracy of nighttime optical flow mainly depends on the knowledge transfer framework during the training phase. In the "3.5 Inference Time'' subsection of the proposed supplementary material, we have compared the runtime and performance of different optical flow backbones that before and after being trained under the proposed framework. In the table below, we further select the latest lightweight optical flow network EMD-S [1] to compare the model size, runtime, and accuracy without and with knowledge transfer:

|              Method               | Model size (M) | Runtime (ms) | EPE  |
| :-------------------------------: | :------------: | :----------: | :--: |
|                GMA                |      5.9       |      79      | 1.48 |
|     GMA w/ common adaptation      |      5.9       |      79      | 0.79 |
|            Transformer            |      18.2      |     273      | 1.43 |
| Transformer  w/ common adaptation |      18.2      |     273      | 0.74 |
|               EMD-S               |      4.5       |      42      | 1.68 |
|    EMD-S w/ common adaptation     |      4.5       |      42      | 0.81 |

​	We have two conclusions. First, model size and runtime are only related to the optical flow backbone. Second, the proposed common space adaptation framework can significantly improve the nighttime optical flow performance of different optical flow backbones. Therefore, the proposed common space adaptation is a plug-and-play training framework that can solve the nighttime optical flow problems.

[1] Deng C, et al. Explicit motion disentangling for efficient optical flow estimation. ICCV, 2023.