Paper: ReBotNet: Fast Real-time Video Enhancement (ICLR-2024)

Reviewer comment (Experiments):
The paper only conducts experiments on the newly proposed datasets. But these two datasets are not larger or more extensive than previous datasets, so I worry that the experiments will not be convincing enough.

Author response:
We have now added results on two more public datasets where our performance is similar to that of RVRT while being much faster and obtaining real-time settings. It can be seen in the table below:

|   Method   |  DVD  | GoPro |
|:----------:|:-----:|:-----:|
| DeepDeblur | 29.85 | 38.23 |
|   DVDNet   | 32.31 | 31.98 |
|     VRT    | 34.24 | 34.81 |
|    RVRT    | 34.30 | 34.92 |
|  ReBotNet  | 34.30 | 34.90 |

We would also like to point out that our curated datasets are comparable in size to the existing datasets while dealing with more challenging cases due to multiple degradations operating on the same video. Also, it emulates live video call and live streaming scenarios unlike existing datasets.

Reviewer comment (Evaluation):
The paper does not seem to have submitted a demonstration video. Judging from the pictures in the paper, the improvement in visual effects is not significant.

Author response:
We provide some demonstration videos of comparison with previous methods as well as direct slider comparison with real videos in [this link](https://anonymous.4open.science/r/iclr_rebuttal2-EB61/Comparison1.mov). Please look at the left hand side to access all the files from the window that pops up from this link. We are worried right now that releasing a lot of demonstration videos might have a problem with protecting our anonymity. We promise to release a link to more demonstration videos after the reviews.

Reviewer comment (Novelty):
"Unlike these works that require compute intensive optical flow, we develop a simple and efficient frame-recurrent setup with low computational overhead." Optical flow calculations do not seem to be necessarily linked to high computational overhead. Refer to "Optical flow estimation using a spatial pyramid network" or "Real-time intermediate flow estimation for video frame interpolation".

Author response:
We apologize to the reviewer for that sentence and thank the reviewer for pointing out those papers. We have removed them in our revised version but highlight that our contributions or work does not have any effect from this as our method takes a completely different approach and provides a working solution for the problem of video enhancement.

Reviewer comment (Experiments):
"A major use case for real-time video enhancement is videoconferencing where the video actually contains the torso/face of the person." This sentence does not seem to be enough to support the paper's experimentation in this scenario only. I am particularly worried that the background of these scenes is static, and the movement of faces is different from ordinary objects, making it impossible to judge the generalization of the proposed method in general scenes. Even though real-time video conferencing is an important requirement, low-overhead video enhancement makes sense for many other users.

Author response:
While we understand the reviewer’s concerns on static scenes, most of the videos in our FullVideo dataset PortraitVideo in fact have dynamic scenes. This is reflected in the demo videos we have now linked (in point 2). In the paper, while we mention  “video actually contains the torso/face of the person”, we do not say that the background is always static. The background can be dynamic and we have included those cases in our datasets as well.

Reviewer comment (Novelty):
Although the author has conducted a thorough analysis of the constructed network in terms of details and motivations, it appears challenging to avoid the fact that the techniques used in this work seem to be readily available. It is hoped that the author can further refine the contributions and strengths of the proposed method in this regard. In other words, there may still be room for improvement in terms of innovation in this work.

Author response:
We highlight our important contributions here:

* proposing the first video enhancement network that can work in real-time settings (more than 30 FPS) and with a good performance.
* curating two new datasets that resemble actual real-world settings like live video calls and video streams. 
* developing an architecture with ConvNext feature extractors and bottleneck Mixers and showing that this configuration has the best latency-performance tradeoff for video enhancement networks.
* ensuring temporal consistency by integrating with other techniques like recurrent training and temporal bottleneck branch. 

To address the particular point of adding more innovation to this work, we would like to point out that our goal for this work was to obtain a strong video enhancement that works in real-time settings rather than solely introducing novelty. Previous works like RVRT provided more than satisfactory performance for our video enhancement but had an issue with latency which was our main motivation as we targetted live video call enhancement. With ReBotNet, we solve this issue and provide the community with a low-compute model that can also perform well; and our goal was accomplished. With that being said, our work is still novel in its architecture design our architecture is completely different from any of the works in the literature for video enhancement.

Reviewer comment (Experiments):
The author uses a direct addition approach to achieve fusion after the two branches, and I'm curious whether the author has tried other methods to better integrate information related to tubelet tokens and image tokens. In other words, I hope the author can provide a brief explanation or analysis of their choice of fusion method.

Author response:
We did explore alternate techniques for the feature fusion, but found out that a simple addition was the most compute effective option. The performance improvements we obtained from more complicated processes were not significant enough for us to consider them as the fusion strategy. We compare the other alternatives we considered in the following table on PortraitVideo dataset:

|   Method  | Latency |  PSNR |
|:---------:|:-------:|:-----:|
|  Addition |  13.15  | 31.25 |
|   Concat  |  16.85  | 31.25 |
| Attention |  17.58  | 31.28 |

It can be seen that although using attention to fuse the features did provide some performance improvement, the model was suboptimal in terms of latency, which is our main focus for this work.

Reviewer comment (Experiments):
The author's introduction to the curated dataset is not sufficiently clear. It is recommended that the author provide a more intuitive comparison between the curated dataset and existing datasets in terms of data quantity, types of data degradation, and whether they are paired, possibly in the form of a table. Some visual examples should also be included in the manuscript. Additionally, the author does not seem to mention whether these two datasets will be made open-source, which has a certain impact on the contribution of this work.

Author response:
We tabulate the difference of our curated datasets with existing datasets in the following:

|                       | PortraitVideo/FullVideo | Existing Datasets |
|:---------------------:|:-----------------------:|:-----------------:|
|      High Quality     |            Y            |         Y         |
|     Live Scenarios    |            Y            |         N         |
|    Focus on Humans    |            Y            |         N         |
| Multiple Degradations |            Y            |         N         |

We will release the dataset by releasing a script to curate and generate the videos from the web similar to TalkingHead dataset.

Reviewer comment (Experiments):
The scenarios targeted by this work are closely related to everyday life. I am quite curious about the performance of the proposed method on some real video data captured using mobile devices. Could the author possibly add relevant experimental results to more comprehensively validate the effectiveness of the proposed method?

Author response:
To make the model work on everyday things like mobile phone videos, we need to scale the data that we train the model on. We consider this our future work. Our aim would be to ensure that the model's training data comprehensively encompasses diverse and representative samples from everyday scenarios, enabling its effective application to diverse content. Currently, our contribution is more towards the network architecture and not towards an generalizable model. In fact, this is also an active field of research in video restoration.

Reviewer comment (Writing):
There appear to be some typographical errors in the manuscript. In the analysis of ReBoNet in Section Five, the author mentions "Table 3 illustrates these results where gray rows correspond to ...", but there doesn't seem to be any gray rows in Table 3. We hope the author can carefully review the manuscript to prevent such situations from occurring.

Author response:
We thank the reviewer for pointing out this error and we have corrected it in the revised version.

Reviewer comment (Theory):
The explanation of the contribution to efficiency and effectiveness is not clear enough. The introduction mentions that the combination of ConvNext and Mixer avoids quadratic complexity and guarantees performance. Does it mean that using Mixer speeds up the process (avoiding quadratic complexity), while ConvNext ensures performance? Is the fundamental reason for the acceleration avoiding quadratic complexity?

Author response:
## 1) Analytical Experiments leading to our design choice:

To comprehend the advantages of encoder design, we started our experiments with a setup altering the encoder design while maintaining a fixed decoder with simple transposed convolution layers. Our objective was to match the performance of these models, aiming to gauge the computational requirements needed to achieve equivalent performance among these different encoders. The performance in terms of PSNR on the Portrait Video dataset and the latency is reported in the table below:

|   Encoder   | Latency (in ms) |  PSNR |
|:-----------:|:---------------:|:-----:|
|   ConvNet   |      86.89      | 31.24 |
|   ConvNext  |    **30.56**    | 31.22 |
| Transformer |      140.58     | 31.25 |
|    Mixer    |    **28.65**    | 31.23 |


We can observe that while Transformers take more inference time and complexity to match ConvNet’s performance while both ConvNext and Mixers are more efficient than their predecessors. 

After considering these insights, our focus shifted towards leveraging the strengths of both ConvNext and Mixer. Combining a convolutional encoder for initial feature extraction with a mixer network in the bottleneck has demonstrated effectiveness. Convolutional layers excel in efficient low-level feature extraction, while mixers, following transformers, showcase strong representative abilities in extracting deep features, as also highlighted in Srinivas et al. (2021). We performed experiments where we slowly introduce Mixer in the bottleneck of ConvNext which can be seen in the following table where the PSNR is reported on PortraitVideo dataset:

|  ConvNext |   Mixer   | Latency (in ms) |  PSNR |
|:---------:|:---------:|:---------------:|:-----:|
| 1,2,3,4,5 |     -     |      30.56      | 31.24 |
|   1,2,3   |    4,5    |    **15.58**    | 31.25 |
|    1,2    |   3,4,5   |      18.79      | 31.25 |
|     1     |  2,3,4,5  |      26.77      | 31.23 |
|     -     | 1,2,3,4,5 |      28.65      | 31.23 |
|    4,5    |   1,2,3   |      36.85      | 31.25 |

Notably, the last row presents an alternative configuration with early mixer layers and bottleneck ConvNext layers, resulting in sub-optimal performance. These analytical experiments were pivotal in finalizing the design choice of ReBotNet.

## 2) Key Takeaways / Influence of ConvNext and Mixers:

Expanding on the tables previously discussed, the key takeaways for the community are: 

* Early ConvNext layers are good low-level feature extractors; better than transformers/mixers.
* Bottleneck Mixers are effective at extracting strong deep features; better than ConvNets/ConvNexts.
* Temporal branch adds temporal consistency and makes inter-frame transitions smoother.
* A combination of ConvNext feature extractors and bottleneck Mixers proves highly effective for video enhancement, offering an optimal tradeoff between latency and performance.

Reviewer comment (Evaluation):
The PSNR and SSIM results are similar to RBRT.

Author response:
## 3) Comparison with RVRT:

Although ReBotNet's performance is on par with RVRT, the primary difference lies in the model's latency. Unlike RVRT, which is unsuitable for real-time applications and remains challenging to scale down while maintaining a favorable latency-to-performance tradeoff (as demonstrated in Table 1), ReBotNet is an entirely different architecture design. This core difference grants ReBotNet the capability for real-time processing, facilitating its effective utilization in live calls and streams, while delivering comparable performance to RVRT.

Reviewer comment (Experiments):
The evaluation of temporal consistency seems lacking in the paper. For instance, although the abstract claims that the second branch improves temporal consistency, there are no experiments to support this result.

Author response:
## 4) Temporal Consistency:

We would like to note that quantitatively evaluating temporal consistency is an actively researched field. Usually, one is able to evaluate temporal consistency by just looking at the video. We attach a [sample predictions link](https://anonymous.4open.science/r/ICLR_rebuttal-4D33/temporalbranch.mov) with and without the temporal branch and one can see that the one with the temporal branch is smoother with transitions in between frames (we recommend the reviewer to focus on the eye region to clearly see the difference).

We quantify temporal consistency via SSIM differences between consecutive frames in the Portrait Video dataset. The configuration with the temporal branch shows lower differences, indicating improved smoothness.

|        Config        | SSIM Difference |
|:--------------------:|:---------------:|
|     Ground Truth     |      0.104      |
| Without Temp. Branch |      0.158     |
|   With Temp. Branch  |      0.124     |

Reviewer comment (Experiments):
I believe the ablation study is not detailed enough. Combining ConvNext and Mixer is a direct and straightforward idea. In the process of parameter tuning, what analytical experiments are worth providing to the community? I expect to see many detailed experiments on this aspect. Can we use other backbones to replace the ConvNext?

Author response:
## 1) Analytical Experiments leading to our design choice:

To comprehend the advantages of encoder design, we started our experiments with a setup altering the encoder design while maintaining a fixed decoder with simple transposed convolution layers. Our objective was to match the performance of these models, aiming to gauge the computational requirements needed to achieve equivalent performance among these different encoders. The performance in terms of PSNR on the Portrait Video dataset and the latency is reported in the table below:

|   Encoder   | Latency (in ms) |  PSNR |
|:-----------:|:---------------:|:-----:|
|   ConvNet   |      86.89      | 31.24 |
|   ConvNext  |    **30.56**    | 31.22 |
| Transformer |      140.58     | 31.25 |
|    Mixer    |    **28.65**    | 31.23 |


We can observe that while Transformers take more inference time and complexity to match ConvNet’s performance while both ConvNext and Mixers are more efficient than their predecessors. 

After considering these insights, our focus shifted towards leveraging the strengths of both ConvNext and Mixer. Combining a convolutional encoder for initial feature extraction with a mixer network in the bottleneck has demonstrated effectiveness. Convolutional layers excel in efficient low-level feature extraction, while mixers, following transformers, showcase strong representative abilities in extracting deep features, as also highlighted in Srinivas et al. (2021). We performed experiments where we slowly introduce Mixer in the bottleneck of ConvNext which can be seen in the following table where the PSNR is reported on PortraitVideo dataset:

|  ConvNext |   Mixer   | Latency (in ms) |  PSNR |
|:---------:|:---------:|:---------------:|:-----:|
| 1,2,3,4,5 |     -     |      30.56      | 31.24 |
|   1,2,3   |    4,5    |    **15.58**    | 31.25 |
|    1,2    |   3,4,5   |      18.79      | 31.25 |
|     1     |  2,3,4,5  |      26.77      | 31.23 |
|     -     | 1,2,3,4,5 |      28.65      | 31.23 |
|    4,5    |   1,2,3   |      36.85      | 31.25 |

Notably, the last row presents an alternative configuration with early mixer layers and bottleneck ConvNext layers, resulting in sub-optimal performance. These analytical experiments were pivotal in finalizing the design choice of ReBotNet.

## 2) Key Takeaways / Influence of ConvNext and Mixers:

Expanding on the tables previously discussed, the key takeaways for the community are: 

* Early ConvNext layers are good low-level feature extractors; better than transformers/mixers.
* Bottleneck Mixers are effective at extracting strong deep features; better than ConvNets/ConvNexts.
* Temporal branch adds temporal consistency and makes inter-frame transitions smoother.
* A combination of ConvNext feature extractors and bottleneck Mixers proves highly effective for video enhancement, offering an optimal tradeoff between latency and performance.