Paper: Progressive Fourier Neural Representation for Sequential Video Compilation (ICLR-2024)

Reviewer comment (Experiments):
Although the quantitative results are listed in details for every video in two settings, overall the proposed method is only tested on two datasets, and both UVG series. More diverse choices would make the results more solid, such as the DAVIS dataset series etc.

Author response:
- **Large-scale dataset (DAVIS50)**

    As reviewer MyBt and H5Ez suggested, we prepare a large-scale sequential video dataset, the Densely Annotation Video Segmentation dataset (DAVIS50). To validate our algorithm and investigate the limitations, we conducted the experiments on 50 video sequences with 3455 frames with a high-quality, high-resolution (1080p). We have included the primary results to show the effectiveness of our PFNR. For simplicity, the table includes the PSNR performances of 10, 20, 30, 40, and 50 video sessions.


    | Method                     | 10    | 20    | 30    | 40    | 50    | Avg.PSNR/BWT |
    |----------------------------|-------|-------|-------|-------|-------|--------------|
    | STL,NeRV                   | 28.92 | 31.10 | 34.96 | 29.35 | 28.87 |  31.41 / -   |
    | WSN, c=30.0 \%             | 19.20 | 20.80 | 23.39 | 21.56 | 21.45 |	21.56 / 0.0 |
    | PFNR, c=30.0 \%, $f$-NeRV2 | 23.14 | 23.14 | 23.14 | 23.14 | 23.08 |	24.22 / 0.0 |
    | PFNR, c=30.0 \%, $f$-NeRV3 | **25.58** | **27.79** | **31.42** | **27.22** | **24.88** |  **27.57** / **0.0** |
    | MTL                        | 23.10 | 23.19 | 24.63 | 22.84 | 23.45 |  24.57 / -   |

    The overall PFNR performances (STL, WSN, PFNR, and MTL) on the DAVIS50 dataset on the UVG8/17 dataset are lower than those on the UVG17 dataset. However, the performance trends of PFNR observed on the DAVIS50 dataset are consistent with the experimental results obtained from the UVG8/17 dataset. Regarding sequential and multi-task learning, the current performance falls short of achieving the target of 30 PSNR. This indicates a need for future work focused on designing model structures capable of sequence learning on large-scale datasets.

Reviewer comment (Presentation):
The compression performance (model size) is not well tested and displayed, especially in the tables. Figure 2 and its paragraph discussed some but is still not clear to link with the values in other tables.

Author response:
- **Compression Performances**

    To support the compression performances (Figure 2), we have prepared the sparsity-wise (c=10%, 30%, 50%, 70%) PSNR performances as shown in supplemental Table 17. Moreover, we have showed the progressive accumulated model capacity of PFNR which is less than 100\% (avg. 95\%), as shown in Fig. 3(b). Thus, the sparse and 8bit-quantized PFNR lead to parameter efficient PSNR performances. The core parts of the bit-wise quantization results are follows as.

    | **PFNR, $f$-NeRV2** | **Sparsity** | **Avg. PSNR** | **Sparsity** | **Avg. PSNR** |
    |---------------------|--------------|---------------|--------------|---------------|
    | **bit=4**           | c=10.0 \%    | 24.79         | c=30.0 \%    | 24.19         |
    | **bit=8**           | c=10.0 \%    | 28.09         | c=30.0 \%    | 29.33         |
    | **bit=16**          | c=10.0 \%    | 28.10         | c=30.0 \%    | 29.36         |
    | **bit=32**          | c=10.0 \%    | 28.10         | c=30.0 \%    | 29.36         |

Reviewer comment (Presentation):
The illustration of the proposed method is relatively limited, e.g. about the details in Fourier space, which might hide the novelty and complexity of the proposed method.

Author response:
- **Illustration of the proposed methods**
    As reviewer MyBt suggested, we have prepared the ablation studies on the proposed FSO to show its effectiveness for sequential neural implicit representations (please see our responses to MyBt's ablation requests).  

    - First, we show the performances of only real part (ignore an imaginary part) in f-NeRV2/3. The PSNR performances of only real part were lower than those of both real and imaginary parts in f-NeRV2/3. We infer that the imaginary part of the winning ticket improves the implicit neural representations.


    - Second, we also investigate the effectiveness of only FSO without Conv.  in f-NeRV2/3. The PSNR performances were lower than FSO with Conv block. Therefore, the ensemble of FSO and Conv improves the implicit representations.


    - Lastly, we investigate the effectiveness of sparse FSO in STL. The sparse FSO boots the PSNR performances in STL. These ablation studies further strengthen the effectiveness of FSO for sequential neural implicit representations.

Reviewer comment (Presentation):
Not many visual comparisons are provided especially in the main paper, and Figure 7 is not specifically explained on their differences and advantages etc.

Author response:
- **Visual Comparisions**

    With limited space and capacity, we provided only important figures and attached more figures with public WSN code in the Appendix at submission. We will upload our revised script soon.
    
    - Supplementary Figure 7 shows the video generation (from t=0 to t=2) with c = 30.0%, showing the PFNR’s output quality and PSNR scores on the UVG17 dataset. At the human recognition level, approximately 30 PSNR provides a little bit of blurred generated images as shown in WSN’s PSNR(29.26); if the PSNR score is greater than 30, we hardly distinguish the quality of sequential neural implicit representations (city session). Thus, our objective is to maintain the sequential neural implicit representation of the approximately 30 PSNR scores. We have achieved this target PSNR score on the UVG17 dataset. 

    - We also show the quantized and encoded PFNR's video generation results in Supplementary Figure 8. We demonstrate that a compressed sparse solution in FP8 (PFNR with $c=30.0 \%$, $f$-NeRV2) generates video representations sequentially without a significant quality drop. Additionally, the results of the FP4 showed that the compressed model can not create image pixels in detail.

Reviewer comment (Writing):
[Minor] Section 3.1 said that Figure 1 is one "possible" structure, while there isn't any other design mentioned in this paper. Maybe there can exist more variants but if not mentioned then Figure 1 is just exactly "our proposed structure" to be clearer.

Author response:
- **Minor Comments**
    - Variant of PFNR

        A straightforward variant of PFNR could depend on the layer position of FSO. In our experiments, the NeRV block layer of FSO determines its variants of PFNR, i.e., f-NeRV2 or f-NeRV3. Another variant of PFNR ignores imaginary parts, as shown in ablation studies. We have stated the importance of imaginary parts in FSO through the ablation studies despite its advantages of reducing parameters for the imaginary parts.

Reviewer comment (Presentation):
[Minor] Figure 1 is overall good but the font and diagram size is a bit small compared to frame images.

Author response:
- **Minor Comments**
    - Figure 1 size and typos

        - As reviewers suggested, we have increased the font and diagram sizes in Figure 1.

        - We have corrected some of the references. We would inspect any typos and references again for the final scripts.

Reviewer comment (Writing):
[Minor] Sometimes the HNerv paper is noted as Nerv (but with the correct 2023 reference).

Author response:
- **Minor Comments**
    - Figure 1 size and typos

        - As reviewers suggested, we have increased the font and diagram sizes in Figure 1.

        - We have corrected some of the references. We would inspect any typos and references again for the final scripts.

Reviewer comment (Theory):
In the abstract and introduction, it is illustrated that since the INRs learn to memorize videos "regardless of data relevancy or similarity", it is hard for them to be generalized to multiple complex data and thus continual learning is important for the models to learn new videos without forgetting previously learned videos. I agree that memorizing videos regardless of data relevancy or similarity limits INRs' efficiency and scalability, but shouldn't this characteristic help an INR to learn multiple unrelated videos compared to those that learn with data relevancy or similarity?

Author response:
- **Question on Motivations**

    - We agree with the reviewer that learning multiple unrelated videos at human and machine-level recognition is essential regardless of data relevancy or similarity.

    - However, our algorithm assumed that even though the low similarity among unrelated videos, they share some representations in common at the neural implicit representation level as shown in Supplementary Figure 6 (Transfer Matrices); the upper triangular looks meaningless Forward Transfer (FWT), however, there might be some shared representations among uncorrelated videos and Figure 3(b) of reused weight (green, at least 10.0 \% for all video session) also support our claims (the remaining weights (90 \%) could be assigned to represent current session video).

    - Therefore, we could conclude that PFNR leverages both aspects of relevancy and irrelevancy through adaptively reused weight selections for sequential video training.

Reviewer comment (Presentation):
The tables are a little hard to interpret.

Author response:
Thank you for your constructive suggestions. Based on your feedback, we would move plots and images to supplementary and increase the portion of interpretable tables in the main script.

Reviewer comment (Writing):
Is it common to use neural implicit representation (NIR)? As I know, many works adopt the term implicit neural representation (INR) rather than NIR.

Author response:
As the reviewer stated, some previous works follow the term, implicit neural representation (INR), but currently, a similar term, neural implicit representation (NIR) is also observed in various tutorials and works.

Reviewer comment (Theory):
The configuration for Fourier transform — Some sparsification protocols for Fourier transform ignore an imaginary part in both Fourier and inverse Fourier transform.

Author response:
We show the performances of only real part (ignore an imaginary part) in f-NeRV2. The PSNR performances of only real part were lower than those of both real and imaginary parts, as shown in above. We infer that the imaginary part of the winning ticket improves the implicit neural representations. We have included this result in our final supplementary script.

    | **UVG8**                                    | **1**     | **2**     | **3**     | **4**     | **5**     | **6**     | **7**     | **8**     | **Avg. PSNR** |
    |---------------------------------------------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|---------------|
    | **PFNR, c=50 \%, $f$-NeRV2**                | **34.46** | **33.91** | **32.17** | **36.43** | **25.26** | **20.74** | **30.18** | **25.45** | **29.82**     |
    | **PFNR, c=50 \%, $f$-NeRV2 w/o imag.** | 34.34     | 33.79     | 32.04     | 36.4      | 25.11     | 20.59     | 30.17     | 25.27     | 29.71         |
    | **PFNR, c=50 \%, $f$-NeRV2**                | **36.45** | **35.15** | **35.10** | **38.57** | **28.07** | **23.06** | **32.83** | **27.70** | **32.12**     |
    | **PFNR, c=50 \%, $f$-NeRV2 w/o imag.** | 35.66     | 34.65     | 34.09     | 37.95     | 25.80     | 21.94     | 32.17     | 26.91     | 31.15         |

Reviewer comment (Experiments):
The performances of baselines in Single Task Learning (STL) — What is the performance of the proposed method with Single Task Learning (STL)? Does the proposed method impact negatively due to the proposed components for continual learning?

Author response:
Our f-NeRV blocks positively impact single-task learning (STL) as follows. Mainly, f-NeRV3 boosts the PSNR performances of NeRV superiorly. This result could impact high-quality image-generation tasks.

    | **UVG8**             | **1** | **2** | **3** | **4** | **5** | **6** | **7** | **8** | **avg. PSNR** |
    |----------------------|-------|-------|-------|-------|-------|-------|-------|-------|---------------|
    | STL, NeRV            | 39.66 | 36.28 | 38.14 | 42.03 | 36.58 | 29.22 | 37.27 | 31.45 | 36.33         |
    | STL, NeRV, $f$-NeRV2 | 39.73 | 36.30 | 38.29 | 42.03 | 36.64 | 29.25 | 37.35 | 31.65 | 36.40         |
    | STL, NeRV, $f$-NeRV3 | **42.75** | **37.65**  | **42.05** | **42.36**  | **40.01**  | **34.21**  | **40.15** |   **36.15**    |  **39.41**              |

Reviewer comment (Experiments):
Comparison to WSN — What is the averaged PSNR and MS-SSIM performance of the proposed method and WSN on UVG8/17 with varying capacity? It is better to visualize in plot of the performance and capacity rather than table because the capacity of the model seems to be important for this setting. I conjecture that table 8 is the good starting point.

Author response:
The following tables show the averaged PSNR and MS-SSIM performances of WSN on UVG8/17 with varying capacity (c=10%, 30%, 50%, 70%). We have included more generated images in the Supplementary.

    | UVG8            | Avg. PSNR | Avg. MS-SSIM |
    |-----------------|-----------|--------------|
    | STL, NeRV       | 36.33     | 0.97         |
    | EWC             | 12.58     | 0.30         |
    | iCaRL           | 26.51     | 0.80         |
    | ESMER           | 21.92     | 0.61         |
    | WSN             | 27.26     | 0.84         |
    | PFNR, $f$-NeRV2 | 29.82     | 0.90         |
    | PFNR, $f$-NeRV3 | 32.12     | 0.93         |
    | MTL             | 30.78     | 0.91         |


    | UVG17           | Avg. PSNR | Avg. MS-SSIM |
    |-----------------|-----------|--------------|
    | STL, NeRV       | 36.94     | 0.97         |
    | EWC             | 11.38     | 0.30         |
    | iCaRL           | 26.67     | 0.65         |
    | ESMER           | 20.95     | 0.62         |
    | WSN             | 26.42     | 0.82         |
    | PFNR, $f$-NeRV2 | 29.36     | 0.90         |
    | PFNR, $f$-NeRV3 | 31.72     | 0.94         |
    | MTL             | 29.42     | 0.90         |

Reviewer comment (Evaluation):
Meaning of upper bound — The proposed method, PFNR, outperforms the upper bound, MLT. What is the reason behind?

Author response:
We observed the two points regarding the results on our PFNR's comparable performances of MTL.
    
    First, as the number of tasks increases, the performances of MTL tend to decrease (see, UVG8, UVG17, and DAVIS50). This is because uncorrelated individual samples could hinder the update gradients of MTL task models' parameters. From this, we speculate that the video sample's statistics (correlations, mean, and variance) should be considered to acquire the best performances of MTL in training.

    Second, we have discussed the main representational difference between WSN and PFNR in Figure 4. The PFNR tends to capture local textures broadly (representations with high frequency and low variances) at lower layer (NeRV3) while while WSN focuses on local objects. This behavior of PFNR could leads to higher PSNR performances. To support our observations, we would conduct an additional representational analysis in terms of the frequency and variance in our future works.

Reviewer comment (Presentation):
Typo — Table 3 shows the result of MS-SSIM while the caption has PSNR.

Author response:
We have revised the typo: Avg. MS-SSIM → Avg. PSNR in Table 3.

Reviewer comment (Writing):
The writing is not so clear. It's hard to learn the connection of Sec 3.1 with other sections.

Author response:
- **Improving writing**

    Based on the reviewer's constructive feedback, we have revised the descriptions (particularly, Sec. 3.1.). The minor revised points are as follows:

    - We have revised the typos: the cited paper (NeRV), "session t".
    - We have redefined the $H_N$ and used to explain the trainable function $f$. For simplicity equation, we ignore the $H_N$.
    - We have moved Algorithm 1 to the same page as Sec. 3.2.

    We would improve the entire script to be more transparent and strengthen our novel points in the final scripts. We will upload our revision soon.

Reviewer comment (Writing):
At the 2nd line of Sec 3, the cited paper is E-NeRV instead of NeRV.

Author response:
- **Improving writing**

    Based on the reviewer's constructive feedback, we have revised the descriptions (particularly, Sec. 3.1.). The minor revised points are as follows:

    - We have revised the typos: the cited paper (NeRV), "session t".
    - We have redefined the $H_N$ and used to explain the trainable function $f$. For simplicity equation, we ignore the $H_N$.
    - We have moved Algorithm 1 to the same page as Sec. 3.2.

    We would improve the entire script to be more transparent and strengthen our novel points in the final scripts. We will upload our revision soon.

Reviewer comment (Writing):
In the caption of Fig 1, the symbol $H_N$ is used without definition and no further usage.

Author response:
- **Improving writing**

    Based on the reviewer's constructive feedback, we have revised the descriptions (particularly, Sec. 3.1.). The minor revised points are as follows:

    - We have revised the typos: the cited paper (NeRV), "session t".
    - We have redefined the $H_N$ and used to explain the trainable function $f$. For simplicity equation, we ignore the $H_N$.
    - We have moved Algorithm 1 to the same page as Sec. 3.2.

    We would improve the entire script to be more transparent and strengthen our novel points in the final scripts. We will upload our revision soon.

Reviewer comment (Writing):
At the line before Eq 3, should it be "session s" instead of "session t"?

Author response:
- **Improving writing**

    Based on the reviewer's constructive feedback, we have revised the descriptions (particularly, Sec. 3.1.). The minor revised points are as follows:

    - We have revised the typos: the cited paper (NeRV), "session t".
    - We have redefined the $H_N$ and used to explain the trainable function $f$. For simplicity equation, we ignore the $H_N$.
    - We have moved Algorithm 1 to the same page as Sec. 3.2.

    We would improve the entire script to be more transparent and strengthen our novel points in the final scripts. We will upload our revision soon.

Reviewer comment (Presentation):
I suggest moving Algorithm 1 to the same page as Sec 3.2, where it is referred.

Author response:
- **Improving writing**

    Based on the reviewer's constructive feedback, we have revised the descriptions (particularly, Sec. 3.1.). The minor revised points are as follows:

    - We have revised the typos: the cited paper (NeRV), "session t".
    - We have redefined the $H_N$ and used to explain the trainable function $f$. For simplicity equation, we ignore the $H_N$.
    - We have moved Algorithm 1 to the same page as Sec. 3.2.

    We would improve the entire script to be more transparent and strengthen our novel points in the final scripts. We will upload our revision soon.

Reviewer comment (Theory):
In Algorithm 1, the operation at line 7 is not differentiable, so how to calculate $\partial L / \partial \rho$?

Author response:
- **Straight-through estimator for $\frac{\partial L}{\partial \rho^{\ast}}$**

    To update the weight score $\rho^{\ast}$, we have used the Straight-through estimator [1,2,3], as stated in Sec. 3.2. The Straight-through estimator estimates the gradients of a function. Specifically, it ignores the derivative of the threshold function (binary mask) and passes on the incoming gradient as if the function were an identity function. So, the threshold function (binary mask) is bypassed in the backward pass.

    [1] Neural networks for machine learning, Hinton, 2012

    [2] Estimating or propagating gradients through stochastic neurons for conditional computation- CoRR2013

    [3] What’s hidden in a randomly weighted neural network?- CVPR2020