Paper: Video Decomposition Prior: Editing Videos Layer by Layer (ICLR-2024)

Reviewer comment (Theory):
This paper proposes to unify various downstream tasks by video decomposition. However, the formulation of the video relighting task in this paper uses one RGB layer to represent the enhanced video and an alpha-layer to represent gamma correction. This formulation does not have any meaning of “decomposition”. On the other hand, this paper carefully designs different decomposition definitions and corresponding constraints for different downstream tasks, but these constraints and decomposition definitions seem to lack a unified and general formulation. Therefore, I felt such kind of “unification” seems too artificial and forced.

Author response:
**Relighting Decomposition:** We understand the reviewer’s concerns regarding the formulation of the relighting task. We would like to emphasize that the decomposition of a frame may not involve only additive decomposition($\alpha$-blending). What we explored in this work (one of our major novel contributions) is to perform multiplicative decomposition on frames of the video sequence. We derived one such multiplicative decomposition utilizing the gamma correction formulation for the frame relighting task. The interesting outcome of this decomposition is we are able to achieve SOTA performance on the relighting task without requiring any task-specific data. Not only that, we are also able to achieve high temporal coherence in the processed relit videos. Such qualitative comparison can be found in videos attached with the webpage(supp.). 

**Unified Framework:** We hope to change the reviewer’s viewpoint about the VDP framework being artificial. We have taken great pains to figure out this method. 

The core of our framework lies in its ability to adapt to different tasks while maintaining a consistent structure. Basically, each task consists of three integral components: prior knowledge (decomposition formulation), motion modeling (via $\alpha$-Net), and appearance modeling (via RGBnet). The **consistent architecture** of $\alpha$-Net and RGBnet establishes a **unified** way of extracting the motion and appearance features from a video, across different tasks. These intermediate motion and appearance features are then utilized with task-specific adaptations that are reflected in the decomposition formulations. Hence, the unifying factor across different tasks is the consistency in the architecture used to extract motion and appearance cues from the videos across all tasks.

Reviewer comment (Novelty):
The proposed formulation is not new (or at least has very few novelty). Overall, the proposed formulation can be summarized as a "data-term + prior" approach, which is a rather common formulation in optimization-based approach for image / video synthesis tasks. Using neural networks as an implicit prior for data term is not new (as also mentioned in the paper - "These approaches have highlighted the importance of formulating a loss function, combined with the optimization of neural network parameters") and the proposed paper extends this idea into other tasks. Yet, the exact formulation designed for each task is dedicated but common (e.g., gamma-correction curves [1] for relighting; alpha-blending [2,3] for video-segmentation, dark-channel prior for dehazing).

Author response:
**Novelty Issues:** We claim novelty on two terms.  First, The relighting task formulation derived from gamma correction is novel for low-lit video enhancements and is unique to our work. Such formulation is not utilized in the work[1] mentioned as a reference by the reviewer. 

Second, The VDP framework’s way of modeling motion and appearance in a video is unique to our work(utilizing $\alpha$-Net and RGBnet). Specifically, to our knowledge, no prior test time optimization technique has utilized FlowRGBs to model the motion of the video, and it is unique to our methodology. 

With regards to the mentioned test-time methods[2,3], they are not at all designed for the VOS task but more for edit propagation tasks. These methods require per-frame masks obtained by image segmentation baselines. So, we do not see any overlap of these techniques with our framework apart from edit propagation tasks.

We also would like to emphasize that maintaining spatio-temporal coherence is a big issue when it comes to tasks like dehazing and relighting, as can be seen from the qualitative results provided in the webpage(supplementary material). Our novelty also stems from the fact that our method can provide high-fidelity results with far superior spatio-temporal coherency. On top of this, we are able to achieve SOTA results for video dehazing and relighting tasks and achieve SOTA among the test-time optimization techniques in UVOS tasks. We hope this clarification will alleviate the reviewer’s concern about the novelty issues.

Reviewer comment (Experiments):
Missing baselines for UVOS tasks: The paper compares with both pre-trained methods and test-optimization methods. For pre-trained methods, there are massive VOS methods [4,5,6] that has far better performance than compared methods in this paper. For test-optimization methods, there are also other video decomposition methods that target this task [2,3].

Author response:
**Missing VOS baselines:** In Table 2, we only compare with other test-time optimization baselines for a fair comparison as the setting of each methodology is exactly the same, i.e., no prior segmentation annotation is given to the model during training. We provide a comprehensive evaluation of the task UVOS task in Table 5 in the appendix because of the limited availability of space in the main paper. We will definitely add the remaining baselines[4,5,6] in there for a more comprehensive comparison. Additionally, if ICLR allows one extra page after acceptance, we would move those comparisons to the main paper.

Reviewer comment (Evaluation):
While the proposed method does not require pre-training, it needs to optimization on each input sequence which takes time. How long does it take for optimizing a video sequence?

Author response:
**Time taken** example: VOS task for one video with 60 frames: the time required to perform inference time optimization for vos task on an average is 40 frames/min. Hence, it would take for a video with 60 frames $\frac{60}{40}$mins or 1min:30 secs. Hopefully, this clarifies the reviewer’s question.

Reviewer comment (Experiments):
Is there a common insight or general guidance on applying this framework to downstream tasks? For example, what if applying this framework to tasks like denoising or super resolution?

Author response:
**General Guidance:** We have explored three major downstream tasks like the VOS, dehazing, and relighting for videos. [1]* utilizes a test-time optimization technique for tasks like denoising, super-resolution, etc. Our framework can be naively extended to such tasks by utilizing similar loss functions; however, we leave this extension for future work.

[1]* `Video dynamics prior’ - Neurips 2023

Reviewer comment (Reproducibility):
In the introduction of Flow similarity loss, the VGG embeddings of masked flow-RGB and those of other layers are used to calculate the cosine similarity. It is unclear how to generate VGG embeddings from masked feature maps for cosine similarity calculation.

Author response:
**2. Flow Similarity Loss:**
We appreciate your query regarding the formulation of our Flow Similarity Loss. This loss is crucial for differentiating the motion of the background from the foreground. The process involves three steps:

- **Step 1:** We take the FlowRGB of the frame at time step $t$ (the process of obtaining FlowRGBs is detailed in Section K of the appendix included in supplementary material) and perform element-wise multiplication with both the foreground mask and the background mask, i.e., 
  - $FlowRGB_{Fg} = FlowRGB^t \odot M^t$
  - $FlowRGB_{Bg} = FlowRGB^t \odot (1-M)^t$

- **Step 2:** We use these obtained $FlowRGB_{Fg}$ and $FlowRGB_{Bg}$ and find their corresponding VGG embeddings. This is done by passing them as input to the VGG encoder (pretrained on ImageNet dataset and denoted as $\phi(.)$) and taking the output of the last layer of the VGG encoder as the embedding vector. Mathematically, we can write these embedding vectors for the foreground and background layer as, 
  - $embed_{Fg} = \phi( FlowRGB_{Fg})$
  - $embed_{Bg} = \phi( FlowRGB_{Bg})$

- **Step 3:** In this final step, we calculate the loss value by taking a cosine value between these embeddings of the foreground and background layer.
  - $L_{FlowSim} = \frac{embed_{Fg} . embed_{Bg}}{|embed_{Fg}| |embed_{Bg}|}$

We minimize the cosine similarity between the VGG embeddings of the background layer and the foreground layer, which translates to the learning of a mask that separates the motion of the background and foreground. We hope this detailed explanation resolves any ambiguities about flow similarity loss.

Reviewer comment (Theory):
In equation (12), the behavior of the reconstruction layer loss resembles that of the L1 loss in the reconstruction loss. To validate the rationale behind the design, it is essential for the paper to elucidate the distinctions between them and elucidate the impact of the reconstruction layer loss.

Author response:
**3. Ablation Study Involving Reconstruction Layer Loss:**
We understand the concerns of the reviewer about the understanding the impact of reconstruction layer loss. We would refer the reviewer to section D.4 of the appendix(part of supplementary) where we perform extensive experiments on the DAVIS-16 dataset to study the impact of each loss term on the final segmentation results. We hope our clarifications have addressed the concerns of the reviewer.

Reviewer comment (Novelty):
The reconstruction loss and warp loss used are very common loss functions and do not offer any novelty.

Author response:
**Novelty issues:**

We understand the concern raised by the reviewer over the novelty of the paper. Here is a more structured view of our model. The core of our framework for any task at hand has three components to it: prior knowledge(decomposition formulation), motion modeling (via $\alpha$-Net), and appearance modeling (via RGBnet). The framework’s way of modeling motion and appearance in a video is unique to our work. Specifically, to our knowledge, no prior test time optimization technique has utilized FlowRGBs to model the motion of the video. 

Second, the utilization of prior knowledge(from gamma correction) to derive a novel decomposition formulation for the relighting task is a unique contribution of our work.
Lastly, we are able to achieve state-of-the-art temporally coherent results utilizing our framework for the task of dehazing and relighting. We would urge the reviewer to please check out the difference in the qualitative video results of our methodology and baselines(We have provided these in an HTML webpage as a part of our supplementary material). To the reviewer’s point, our work derives the novelty from the above-mentioned points as opposed to the reconstruction or warping loss. We hope our clarification will alleviate the novelty concerns of the reviewer.

Reviewer comment (Novelty):
The method is still not sufficiently general. For example, for the task of video segmentation, additional loss functions need to be designed as constraints. The three task-specific losses mentioned in Section 3.2 are the only ones provided. From this perspective, it is difficult to see what this framework proposed in the paper brings to this community.

Author response:
**What is the consistency across tasks:**

We thank the reviewer for pointing out the difficulty in understanding the consistency between the tasks. Hopefully, our clarification about it will alleviate this difficulty. 

The core of our framework lies in its ability to adapt to different tasks while maintaining a consistent structure. Basically, each task consists of three integral components: prior knowledge (decomposition formulation), motion modeling (via $\alpha$-Net), and appearance modeling (via RGBnet). The **consistent architecture** of $\alpha$-Net and RGBnet establishes a **unified** way of extracting the motion and appearance features from a video across different tasks. These intermediate motion and appearance features are then utilized with task-specific adaptations that are reflected in the decomposition formulations. Hence, the unifying factor across different tasks is the consistency in the architecture used to extract motion and appearance cues from the videos across all tasks.

We would put more effort into revamping the introduction section of our manuscript to emphasize this point of consistency our framework's architecture brings across this variety of tasks.

Reviewer comment (Experiments):
The comparison results of the experiments are not that fair. For instance, in Table 3, the comparison is made with the latest algorithm, CG-IDN (a 2021 algorithm). There are many dehazing algorithms that could be compared, including single-image hazing algorithms with stability processing (references [1][2]).

Author response:
**Additional Experiments:**

We provide more comparisons with the baseline suggested by the reviewers for the dehazing task. We utilize the MSBDN as an image dehazing baseline. Please find the findings below.
| Metrics | Blind[1] | DVP[2] | Ours |
|----------|----------|----------|----------|
| PSNR|22.98| 23.05 | 24.83|
|SSIM| 0.877 | 0.882 | 0.916|

Reviewer comment (Experiments):
Table 2 lacks many baselines. Looking at the official website of DAVIS2016, the best baseline achieves an IOU score of over 82. Why wasn't this paper compared against that?

Author response:
**Baseline comparison for UVOS Task:**

We thank the reviewer for raising this point. In Table 2, we only compare with other test-time optimization baselines for a fair comparison as the setting of each methodology is exactly the same, i.e., no prior segmentation annotation is given to the model during training. We provide a comprehensive evaluation of the task UVOS task in Table 5 in the appendix because of the limited availability of space in the main paper. We would add the results to the main paper if ICLR allows one extra page in the final camera-ready paper.

Reviewer comment (Theory):
Sometimes it is necessary to introduce additional prior knowledge, such as for the task of dehazing, where the effectiveness is actually limited.

Author response:
**Additional Prior knowledge for the dehazing task:**

We agree with the reviewer that additional knowledge about the scene can improve the efficacy of our methodology. However, it is very difficult to acquire hazy video and clean video pairs in a real-world setting. Hence, we formulated a test time optimization technique that does not rely on task-specific data for performing the dehazing.