Paper: STREAM: Spatio-TempoRal Evaluation and  Analysis Metric for Video Generative Models (ICLR-2024)

Reviewer comment (Evaluation):
While I think it is useful to have metrics that focus on different aspects of the generation, I would imagine that while developing a method, most of the time it is more helpful to have a single metric to look at to judge progress (if I'm developing a new method and my newest change to the method improves STREAM-F but hurts STREAM-T, is it a good modification or not. Thus, I think the proposed STREAM metrics will be helpful, but will fail to actually replace FVD. If the authors could find a good way to combine their measurements into a single number (e.g. akin to the F1-score to combine P&R), I think the paper would have more impact.

Author response:
**R4-W1. Find a good way to combine their measurements into a single number (e.g., F1-score):**  
Please see our GC2 in general comments.  
&ensp;

Reviewer comment (Writing):
I find it confusing that the authors introduce the term "STREAM-S": Its name implies that it's one of the metrics that is being introduced (it follows the same naming convention), but it actually isn't. I'm absolutely unclear what that term actually denotes. I think the paper would improve in clarity if that term would be removed and instead clearly state that there are 3 new metrics that are being introduced (STREAM-T, STREAM-F, STREAM-D).

Author response:
**R4-W3. Clearly state that there are 3 new metrics (STREAM-T, F, and D):** STREAM-S stands for STREAM-Spatial quality, composed of STREAM-F and STREAM-D. As suggested by the reviewer, we will make it clearer that we are introducing 3 new metrics (STREAM-T, F, and D).  
&ensp;

Reviewer comment (Writing):
Section 2.1: "Let real and fake video datasets as"  => ... be denoted as...

Author response:
**(1) “Let real and fake video datasets as X and Y=> be denoted as”:**  
We have revised this part as ”Let real and fake video datasets be denoted as X and Y, respectively”.

Reviewer comment (Writing):
Section 2.2: we utilize SwAV (Caron et al., 2020), a proficient image embedding network => SwAV is an algorithm, not a network

Author response:
**(2) “SwAV is an algorithm, not a network”:**  
As advised by the reviewer, we have modified this part to “we utilize a proficient image embedding network proposed by Caron et al. (2020)”.

Reviewer comment (Novelty):
"In the image generation task, Kynka¨anniemi et al. ¨ (2019) have proposed precision and recall (P&R) to separately evaluate the fidelity and diversity of the generated image quality." => I think this was first proposed by Sajjadi et al, (NeurIPS 2018), Assessing Generative Models via Precision and Recall.

Author response:
**(3) “P\&R is first proposed by Sajjadi et al, (NeurIPS 2018)”:**  
We have clarified this part by explicitly changing it to “improved precision and recall (P\&R)”.

Reviewer comment (Writing):
Section 2.4: "If we define a sphere with centered at sample point"  => I don't understand this sentence

Author response:
**(4) “If we define a sphere with centered at sample point => I don’t understand this”:**  
This part explains the issues encountered when applying the P\&R support estimation method straight-forwardly. For better clarity, we explicitly mentioned in the main text as “In the process of P\&R support estimation” and provided a more detailed discussion in Appendix A.4.

Reviewer comment (Experiments):
What is the reason for not comparing against VIS?

Author response:
**R4-W2 & Q2. Compare with Video Inception Score (VIS) (Table 1, 2, 5, Figure 20 and 21):**  
We have added VIS for all the experiments based on the UCF-101 dataset. Additionally, we have compared STREAM with VIS under various temporal distortion scenarios in Figure 21 (Appendix A.15). The experimental result shows that VIS cannot discern different levels of temporal distortions such as shuffling video frames within the video or replacement of video with static scenes (which are identical settings to Figure 3 and 4).  
&ensp;

Reviewer comment (Experiments):
STREAM-T uses a histogram comparison. How do binning sizes affect the outcome?

Author response:
**R4-Q3. How do bin sizes affect the outcome (Figure 17 and 18):**  
STREAM-T exhibits robustness across various types of visual noise regardless of bin size. We have added an ablation study on histogram bin size in **Figure 17 and 18 (in Appendix A.11)**. Furthermore, in the presence of various types of temporal distortions, STREAM-T demonstrates consistent evaluations, except when the bin size becomes extremely small. We have selected the bin size ($= 50$) that exhibits the most optimal performance.  
&ensp;

Reviewer comment (Reproducibility):
Will the authors provide source code for implementing STREAM? It seems nontrivial to implement.

Author response:
**R4-Q4. Provide source code for implementing STREAM:**  
We have already provided the link of our code at the end of Abstract. Additionally, we are planning to add an example page that demonstrates how the current implementation can be easily and simply executed with just a few lines of code.  
&ensp;

Reviewer comment (Evaluation):
One of the main selling points of the FVD metric was it penalizes blur phenomena significantly higher than noise phenomena. This was a useful property because it correlates well with human vision. Additionally, the video generation methods tend to produce blurry samples which would score higher on traditional metrics like SSIM and PSNR. I would want to see how it performs on blurry videos (apply Gaussian blur).

Author response:
**R3-Q2. Check STREAM metric with gaussian blur (Figure 19):**  
Thanks for pointing this out. Similar to the usefulness of FVD in practice, We have observed STREAM-S shows significant sensitivity toward Gaussian blur phenomena. For detailed experimental results, please refer to our **Figure 19 (in Appendix A.12).**  
&ensp;

Reviewer comment (Experiments):
What would be the results if the video is reversed temporally and evaluated? Please do the same for flipping the videos spatially (take mirror images of the frames) and lastly, run the evaluation for both spatially and temporally flipped video sequences. I would like to see an evaluation of all these three scenarios before making my final decision.

Author response:
**R3-Q3.** We have conducted the experiments mentioned by the reviewer in **Table 5 (in Appendix A.14)**. Thank you for your comments. Your feedback greatly improved the understanding of our work. We will divide the reviewer’s question into three parts for clarity:  

**(1) Evaluation on temporally reversed videos:**  
First, it is indeed a valid point that STREAM-T does not assign low values when the frames of a video are completely reversed. Given the nature of our metric, which focuses on evaluating continuity and consistency through frequency analysis, STREAM-T remains agnostic to the direction of time. However, this unique characteristic allows STREAM-T to be sensitive not only to situations where only a portion of the video sequence is reversed but also to various types of temporal inconsistencies.  

However, it is important to note that the generation of a completely reversed video  is an uncommon scenario, and in such cases all the metrics (STREAM-T, VIS, and FVD) may not provide optimal responses.  Still, other than this rare case, STREAM demonstrates its effectiveness in detecting partially reversed segments within videos that are otherwise temporally natural (Table 6 in Appendix A.14). For example, when the video is entirely reversed on the  temporal axis, FVD shows a modest increase from 14.38 to 24.93. This change, while noticeable, is relatively small compared to significant fluctuations (such as those in the hundreds) observed in other cases (Figure 5 and 6 in Section  3.2.1). Thus, in practice, this minor variation would make it difficult to conclusively determine if FVD is significantly impacted by this aspect alone.  

Lastly, typical video data, including datasets like CATER, UCF-101, Kinetics, Taichi, and BAIR Robot Pushing, often features repetitive scenes, such as rotations, repetitive exercises, and cooking. In such practical scenarios, STREAM-T is effective in evaluating inconsistencies in continuity and consistency of temporal movements. This aspect, acknowledging the limitations and strengths of STREAM-T in handling various temporal dynamics, will be incorporated into the limitations section.  

**(2) Evaluation on spatially flipped videos:**  
Spatially flipped videos, being the mirror image of each video frame, possess highly realistic spatial and temporal quality. Through the experiment, we have observed that STREAM and VIS remains unchanged, while FVD slightly increased from 14.38 to 36.07.  

**(3) Evaluation on temporally reversed and spatially flipped videos:**  
This experiment simply applied the above two cases simultaneously, and the metrics should provide scores indicating spatially and temporally realistic results. From the results, VIS and STREAM consistently exhibited scores identical to the reference score, whereas FVD showed an increased value from the reference score (14.38) to 40.01.

Reviewer comment (Evaluation):
Using P&R to compute fidelity and diversity of fake videos is reasonable, but why to use amplitude at frequency of 0 as sample points? I would like to see more explanations from the authors.

Author response:
**R2-W2. Reason for using amplitude at frequency 0 when computing STREAM-S:**  
In STREAM-S, we use the amplitude at frequency 0 (i.e., mean amplitude) calculations for two main reasons. First, the mean amplitude encapsulates the average signal information of a video, offering a comprehensive overview of its spatial characteristics. Second,  including amplitudes across all frequencies renders P\&R calculation inaccurate.  

More specifically, P\&R is designed to approximate the support of real and fake distributions, typically using the union of spheres centered on data points with radii equal to the distance to the  $k$-nearest neighbor. These spheres are expected to cover at least $k$ data points (or image features) and collectively represent the population data support. However, videos, unlike images, are composed of multiple image frames. For instance, a 16-frame video yields 16 image features obtained by an embedding network. Applying FFT along the temporal axis, we get 8 amplitude features for each video. In this context, using P\&R directly on videos might result in inaccurate data support estimation, as a single sphere may not even encompass all data points from one video. Naturally, in such a scenario, the approximated data support may fail to represent the full scope of video data. To address this, we use the mean amplitude, which represents the average feature of a video, as a single data point. This approach ensures a more accurate and representative approximation of real and fake video data support, enhancing the precision of STREAM-S measurements without limiting it to the number of video frames as shown in our experiments.  
&ensp;

Reviewer comment (Evaluation):
As video quality is very subjective, a systematic evaluation by human raters is required to compare with current method FVD.

Author response:
**R2-W3. Compare FVD with STREAM in human evaluation (Appendix A.6):**  
Please see **Section 3.2.2 and Appendix A.6.** We have already compared FVD with STREAM in our human evaluation experiment. For the reviewer’s convenience, summarizing here, we measured Spearman’s correlation between human-evaluation and metrics (STREAM and FVD) in terms of video realism and temporal naturalness. The results show that STREAM has a higher correlation with both aspects evaluated by humans than FVD (i.e.,corr(realism, STREAM) = 0.9, corr(realism, FVD) = 0.7, corr(temporal naturalness, STREAM) = 0.6, corr(temporal naturalness, FVD) = 0.5).  
&ensp;

Reviewer comment (Evaluation):
The difference in the style of real and fake video datasets may have a huge effect on the result. For example, if real videos tend to have fast changes, the frequency may be concentrated on the high frequency, and the curve fitted may have a higher skewness. In such case, if the curve fitted by fake videos is less steep (fake videos change slowly), skewness may be small, and even if the fake video is of high quality, STREAM-T may be small.

Author response:
**R2-W4. Difference in the style of real and fake video datasets may have a huge effect on the result (Table 5):**  
Similar to our response in R2-W2, the objective of video generative models is to mimic the spatial and temporal characteristics of real dataset by estimating the real data distribution. Therefore, when there are “differences in style of real and fake video datasets”, an optimal evaluation metric should be able to discern this difference. Therefore, “when the real videos tend to have fast changes while fake videos change slowly”, STREAM-T should give a small score to reflect this. As suggested by the reviewer, we have compared the real videos with fast FPS and generated videos with slow FPS using STREAM-T **(Table 4 in Appendix A.13)**. In the result, STREAM-T has a decreased value when the generated video has a relatively low FPS compared to the real video, which shows the effectiveness of STREAM-T in capturing the temporal difference.

Reviewer comment (Experiments):
The experimental section of the paper is relatively weak, being limited to a few older baseline models. Authors should focus more on current open-source Text-to-video models, which would provide more convincing results.

Author response:
We would like to clarify that the unconditional video generative models compared in Section 3.2.2 and 3.2.3 include various recent open-source video models, such as **PVDM (2023), MeBT (2023), TATS (2022), and DIGAN (2022) [1,2,3,4]**. Given the inherent nature of evaluation, achieving a fair comparison between video generation models relies on the use of open-source pretrained models. We kindly request your understanding that many of the recent models showing state-of-art performance often do not make these models publicly available. Please see our answer in R1-W2 for the new experiments we have conducted.  
&ensp; - [1] Yu, Sihyun, et al. "Video probabilistic diffusion models in projected latent space."  Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.  
&ensp; - [2] Yoo, Jaehoon, et al. "Towards End-to-End Generative Modeling of Long Videos with Memory-Efficient Bidirectional Transformers." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.  
&ensp; - [3] Yu, Sihyun, et al. "Generating videos with dynamics-aware implicit generative adversarial networks." arXiv preprint arXiv:2202.10571 (2022).  
&ensp; - [4] Ge, Songwei, et al. "Long video generation with time-agnostic vqgan and time-sensitive transformer." European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2022.  
&ensp;