Paper: Unveiling Temporal Telltales: Are Unconditional Video Generation Models Implicitly Encoding Temporal Information? (ICLR-2024)

Reviewer comment (Experiments):
Experiments should encompass multiple baselines to bolster the claim that current unconditional video generation methods implicitly encode temporal information.

Author response:
(Q1 & W1-1) Why not use MoCoGAN-HD and StyleGAN-V in the preliminary experiment? This would avoid reproducing MoCoGAN-256. / Experiments should encompass multiple baselines to bolster the claim that current unconditional video generation methods implicitly encode temporal information.

(Response – Q1 & W1-1) The results in the preliminary section can be extended if we bring MoCoGAN-HD and StyleGAN-V results in the experiment section (Table. 2, 3, 4). However, we only presented the MoCoGAN results to avoid redundancy. Nevertheless, we agree that it would be better bolstered if we tested on diverse models (i.e., Diffusion-based model). As suggested by another reviewer, we have tested on recent diffusion-based unconditional video generation model to show the problem of implicit temporal encoding does not occur only in the GAN-based models but also in the diffusion-based model. Similar to the GAN-based model, the diffusion-based model also showed the existence of the problem by being able to temporally classify the generated videos. We have added the section in the revised appendix for this experiment (45.55% in Sky-Timelapse and 19.66% in the UCF-101).

Reviewer comment (Experiments):
The constructed dataset used for training the temporal classifier appears overly homogenous since the clips are sampled from a single video clip of FaceForensics, Sky–Timelapse, or UCF-101.

Author response:
(W1-2&3 & Q2) The constructed dataset used for training the temporal classifier appears overly homogenous since the clips are sampled from a single video clip of FaceForensics, Sky–Timelapse, or UCF-101. / The labels for certain frames seem less meaningful due to the frame's presence in various clips at different positions, arising from the repeated random sampling during the construction of the temporal classification dataset. / Can you visualize some real and generated video frames used in the preliminary experiment? This could enhance clarity.

(Response – W1-2&3 & Q2) As we fix the content noise vector while varying the motion noise vector, the generated videos consist of frames that look similar to another frame located in another temporal location and are also overly homogenous. With these generated videos, the CNNs are able to classify the temporal location of the frames better than the real-world videos. To mitigate the same dataset construction, we have meticulously designed the real-world dataset for temporal classification. We have added some samples for temporal classification in the revised appendix.

Reviewer comment (Experiments):
The labels for certain frames seem less meaningful due to the frame's presence in various clips at different positions, arising from the repeated random sampling during the construction of the temporal classification dataset.

Author response:
(W1-2&3 & Q2) The constructed dataset used for training the temporal classifier appears overly homogenous since the clips are sampled from a single video clip of FaceForensics, Sky–Timelapse, or UCF-101. / The labels for certain frames seem less meaningful due to the frame's presence in various clips at different positions, arising from the repeated random sampling during the construction of the temporal classification dataset. / Can you visualize some real and generated video frames used in the preliminary experiment? This could enhance clarity.

(Response – W1-2&3 & Q2) As we fix the content noise vector while varying the motion noise vector, the generated videos consist of frames that look similar to another frame located in another temporal location and are also overly homogenous. With these generated videos, the CNNs are able to classify the temporal location of the frames better than the real-world videos. To mitigate the same dataset construction, we have meticulously designed the real-world dataset for temporal classification. We have added some samples for temporal classification in the revised appendix.

Reviewer comment (Presentation):
An invalid figure reference in the last paragraph of Section 3 suggests a missing figure in the manuscript.

Author response:
(W2-1 & 2) Figure 2 lacks an introduction of f_{temp} in the caption, and the caption references an ImageNet pre-trained model not presented in the figure. / An invalid figure reference in the last paragraph of Section 3 suggests a missing figure in the manuscript.

(Response – W2-1&2) Thank you for identifying places where additional explanations were needed and for pointing out typos. We have added the explanation in Figure 2 for the f_{temp} in the revised version. The ImageNet pre-trained model that we utilized is inside the torchvision package provided by PyTorch and there is no exact reference for this. However, we did add that we utilized ImageNet pre-trained ResNet-18 in the caption of Figure 2. The invalid figure reference in the last paragraph of Section 3 was supposed to refer to Figure 2 in the manuscript, we have corrected the reference in the revised version.

Reviewer comment (Reproducibility):
The training quality of the reproduced MoCoGAN-256 appears suboptimal. Tables 2, 3, and 4 reveal an extremely high FVD value for MoCoGAN with 256x256 compared to the other high-resolution video generation baselines.

Author response:
(W3) The training quality of the reproduced MoCoGAN-256 appears suboptimal. Tables 2, 3, and 4 reveal an extremely high FVD value for MoCoGAN with 256x256 compared to the other high-resolution video generation baselines.

(Response – W3) MoCoGAN was the first unconditional video generation model that disentangled the content and motion noise vector for generating videos. It is important to note that the primary focus of MoCoGAN was not centered on generating high-resolution videos such as 256x256 but on generating 64x64 videos. In the MoCoGAN paper, they experimented on the Weizmann Action dataset and UCF-101 resized to 64x64. Therefore, the MoCoGAN-256 could seem suboptimal when compared to MoCoGAN-HD and StyleGAN-V, the methods aimed at generating 256x256 videos.

Reviewer comment (Experiments):
The validity of the negative gradient provided by the temporal classifier during training needs reconsideration. The temporal classifier is trained using frames from different videos, which is different from the practice in the preliminary experiment and the evaluation stage that constrains the training frames to have the same content.

Author response:
(W4) The validity of the negative gradient provided by the temporal classifier during training needs reconsideration. The temporal classifier is trained using frames from different videos, which is different from the practice in the preliminary experiment and the evaluation stage that constrains the training frames to have the same content.

(Response – W4) A batch of generated videos for training the temporal classifier, different from the videos utilized for training the discriminator, is generated with fixed content noise as noted by the reviewer. We added an additional explanation in the revised version.

Reviewer comment (Experiments):
The absence of recent diffusion models for video generation (Luo et al., 2023; Yu et al., 2023; Harvey et al., 2022) in the experiments diminishes the contribution of this work.

Author response:
(W5) The absence of recent diffusion models for video generation (Luo et al., 2023; Yu et al., 2023; Harvey et al., 2022) in the experiments diminishes the contribution of this work.

(Response – W5) We admit that the problem has to be general across different backbones (GAN-based model and Diffusion-based model). From the suggested methods, we have tested PVDM[1] as they have an open-source implementation with the author-provided checkpoints. Due to the time constraint of the discussion period, we could not train the PVDM and PVDM + ours to demonstrate the effectiveness of our method. However, we did an experiment to see if the implicit temporal encoding is also occurring in the diffusion-based model. For the experiment, we generated the videos in Sky-Timelapse and UCF-101 with the checkpoint provided by the author. Consistent with our experiments in the manuscript, we have fixed the content noise vector while varying the motion noise vector, and maintain other experimental settings identical to the original experiment. The table below shows the temporal classification accuracy of the generated videos. The results demonstrate that implicit temporal encoding also occurs in the diffusion-based model. We have added the section in the appendix in the revised version with the visual samples of the generated videos that were used in the experiment.

| Dataset | Sky-Timelapse | UCF-101 |
| --- | --- | --- |
| Real-World Videos | 9.47% | 15.72% |
| Generated Videos (PVDM) | 45.55% | 19.66% |

Reviewer comment (Presentation):
Additional insights on designing architectures that "do not necessitate classification" in the discussion section would be beneficial.

Author response:
(W6) Additional insights on designing architectures that "do not necessitate classification" in the discussion section would be beneficial.

(Response – W6) We sincerely appreciate the reviewer for raising this point. This is definitely a go-to way for future research. As stated in the discussion section, employing a classification model may pose challenges when generating extensively long videos (i.e., 1-hour or longer videos). We may try to make the classification model to learn the relative temporal location, however, this may be suboptimal or even not work properly. Unfortunately, we do not currently have any insights on architecture that do not necessitate classification. If possible, it would be a great honor to hear back the thoughts on the potential architecture construction.

Reviewer comment (Experiments):
Why not use MoCoGAN-HD and StyleGAN-V in the preliminary experiment? This would avoid reproducing MoCoGAN-256.

Author response:
(Q1 & W1-1) Why not use MoCoGAN-HD and StyleGAN-V in the preliminary experiment? This would avoid reproducing MoCoGAN-256. / Experiments should encompass multiple baselines to bolster the claim that current unconditional video generation methods implicitly encode temporal information.

(Response – Q1 & W1-1) The results in the preliminary section can be extended if we bring MoCoGAN-HD and StyleGAN-V results in the experiment section (Table. 2, 3, 4). However, we only presented the MoCoGAN results to avoid redundancy. Nevertheless, we agree that it would be better bolstered if we tested on diverse models (i.e., Diffusion-based model). As suggested by another reviewer, we have tested on recent diffusion-based unconditional video generation model to show the problem of implicit temporal encoding does not occur only in the GAN-based models but also in the diffusion-based model. Similar to the GAN-based model, the diffusion-based model also showed the existence of the problem by being able to temporally classify the generated videos. We have added the section in the revised appendix for this experiment (45.55% in Sky-Timelapse and 19.66% in the UCF-101).

Reviewer comment (Presentation):
Can you visualize some real and generated video frames used in the preliminary experiment? This could enhance clarity.

Author response:
(W1-2&3 & Q2) The constructed dataset used for training the temporal classifier appears overly homogenous since the clips are sampled from a single video clip of FaceForensics, Sky–Timelapse, or UCF-101. / The labels for certain frames seem less meaningful due to the frame's presence in various clips at different positions, arising from the repeated random sampling during the construction of the temporal classification dataset. / Can you visualize some real and generated video frames used in the preliminary experiment? This could enhance clarity.

(Response – W1-2&3 & Q2) As we fix the content noise vector while varying the motion noise vector, the generated videos consist of frames that look similar to another frame located in another temporal location and are also overly homogenous. With these generated videos, the CNNs are able to classify the temporal location of the frames better than the real-world videos. To mitigate the same dataset construction, we have meticulously designed the real-world dataset for temporal classification. We have added some samples for temporal classification in the revised appendix.

Reviewer comment (Evaluation):
What is the reason for using distinct videos for evaluating temporal accuracy and FVD computation? Why not use the temporal classifier from the training process to evaluate temporal classification accuracy?

Author response:
(Q3) What is the reason for using distinct videos for evaluating temporal accuracy and FVD computation? Why not use the temporal classifier from the training process to evaluate temporal classification accuracy?

(Response – Q3) The FVD metric measures the feature distance similarity between the real-world videos and generated videos. Recognizing the suboptimal quality of early-stage generated videos, we employed a fully trained unconditional video generation model for generating images for measuring FVD and temporal accuracy. 
As for the temporal classifier, we did not use the one from the training as the reviewer stated. While the temporal classifier was designed to accurately classify the temporal location of frames during training, the inclusion of the Gradient Reversal Layer (GRL) before the temporal classifier may have harmed its classification ability. To ensure precise measurement of the temporal accuracy we re-trained the temporal classification model.
To make a fully functional temporal classification, one would need a sufficient amount of data. In this work, we generated 2,048\times0.8 videos for training.

Reviewer comment (Experiments):
Can you provide qualitative comparisons on UCF101? This would provide additional insights.

Author response:
(Q4) Can you provide qualitative comparisons on UCF101? This would provide additional insights.

(Response – Q4) The quality of the generated UCF-101 videos hampers from comparing the results visually. Prior works also show the Sky-Timelapse and FaceForensics for qualitative comparison as done in our paper. However, we added some UCF-101 results in the revised appendix to provide additional insights to the reviewer.

Reviewer comment (Evaluation):
Can you provide some failure cases of video generation with GRL?

Author response:
(Q5) Can you provide some failure cases of video generation with GRL?

(Response – Q5) We acknowledge the importance of presenting failure cases for advancing research in this field. However, we have not observed any failure cases while training. However, a potential failure scenario arises when the temporal classifier struggles to classify temporal location during training. Specifically, if the generated videos lack temporal information (i.e. if the generator is not implicitly encoding the temporal information) it could result in 1) a misclassification of videos in terms of temporal location and consequently 2) a failure of the Gradient Reversal Layer (GRL) to fulfill its intended function.

Reviewer comment (Experiments):
Lack of comparison with recent methods, e.g., [1, 2, 3]. These methods seem to have much lower FVD than the proposed methods on UCF101 dataset.

Author response:
For comparison, we have selected a recent GAN-based model in the unconditional video generation field that incorporates separate content and motion noise vectors. Also, we highlight that we are not claiming to have achieved state-of-the-art performance compared to recent works. Rather, we emphasize that by neglecting implicitly encoded temporal information, we obtain a better FVD score while maintaining similar qualitative video generation results. However, we admit that experimenting with diffusion-based unconditional video generation models enriches our paper by indicating that implicit temporal encoding occurs not only with the GAN-based models but also with diffusion-based models.
Due to the time constraint of the discussion period, we were unable to train the diffusion-based model and diffusion-based model + ours. Consequently, we focused on testing whether the generated videos from the trained diffusion-based model are temporally classifiable or not. We searched for a diffusion-based model that has open-source implementation and the checkpoint provided by the author. We were not able to find the right fit in the reviewer’s suggested methods therefore we present the experiment done on another reviewer’s suggested method, PVDM[1]. In line with our original experiments, we have fixed the content noise and varied the motion noise in PVDM. The generated videos of Sky-Timelapse and UCF-101 with PVDM do have higher temporal classification accuracy when compared to the real videos in Sky-Timelapse and UCF-101 as shown in the below table.

| Dataset | Sky-Timelapse | UCF-101 |
| --- | --- | --- |
| Real-World Videos | 9.47% | 15.72% |
| Generated Videos (PVDM) | 45.55% | 19.66% |

Although the difference in test accuracy of UCF-101 may seem negligible, we argue that an improvement in the quality of the generated UCF-101 videos could correspond to an increase in temporal accuracy. We added qualitative and quantitative results in the appendix of the revised version as this could give insights for research in diffusion-based generation models.

Reviewer comment (Novelty):
I am not able to see why encoding temporal information in generated videos is a major problem that prevents GAN-based methods to generate high quality videos.

Author response:
As implied in the manuscript, our main statement is that videos generated with current unconditional video generation methods have temporal information encoded as the reviewer said and our method ensures temporal information to be not encoded in generated videos. More importantly, we are not proposing additional loss terms or modules aiming at generating better quality videos. We demonstrated that by neglecting such temporal information encoding when training the generation model, we achieved better FVD scores while showing similar visual quality with the original works as shown in Figure 3 (FaceForensics and Sky-Timelapse) and in the revised appendix (UCF-101).

Reviewer comment (Presentation):
There are "??" in this paper.

Author response:
Thank you for catching the minor mistake, the correct reference can be seen in the revised version. 
While acknowledging the challenge of precisely defining realness, we attempted to characterize it as video possessing similar attributes to those in real-world videos and with how humans perceive the temporal location within videos. To quantify this aspect, the commonly used FVD metric could be a suitable choice. However, we propose considering a temporal classification metric as well. This metric aligns with human behavior, connecting the way humans perceive videos and classify temporal location with the notion of realness.

Reviewer comment (Writing):
The authors claim that they investigate "meaning of ‘realness’ in the video generation models" in the abstract. However, I am having a hard time finding how the meaning of 'realness' connects to the proposed method.

Author response:
Thank you for catching the minor mistake, the correct reference can be seen in the revised version. 
While acknowledging the challenge of precisely defining realness, we attempted to characterize it as video possessing similar attributes to those in real-world videos and with how humans perceive the temporal location within videos. To quantify this aspect, the commonly used FVD metric could be a suitable choice. However, we propose considering a temporal classification metric as well. This metric aligns with human behavior, connecting the way humans perceive videos and classify temporal location with the notion of realness.

Reviewer comment (Theory):
Why encoding temporal information in generated videos is a major problem that prevents GAN-based methods to generate high quality videos?

Author response:
As implied in the manuscript, our main statement is that videos generated with current unconditional video generation methods have temporal information encoded as the reviewer said and our method ensures temporal information to be not encoded in generated videos. More importantly, we are not proposing additional loss terms or modules aiming at generating better quality videos. We demonstrated that by neglecting such temporal information encoding when training the generation model, we achieved better FVD scores while showing similar visual quality with the original works as shown in Figure 3 (FaceForensics and Sky-Timelapse) and in the revised appendix (UCF-101).

Reviewer comment (Evaluation):
It is not clear why encoding the temporal information inside the frame would lead to the video quality degradation. Some empirical / theoretical explanation could be useful to provide further insights.

Author response:
The temporal information inside the frame does not lead to visual quality degradation of the video. To put it in another way, the temporal information does affect the FVD but not the visible quality of the video as can be seen in Figure 3 (FaceForensics and Sky-Timelapse) and in the revised appendix (UCF-101). In terms of FVD, we hypothesize that it could degrade due to temporal information encoding as each frame from a generated video has additional information that is not essential/needed in case of the real-world videos.

Reviewer comment (Writing):
The argument that since CNN is inspired from humans, then they should not be able to detect the temporal signal embedded in the generated frames is not necessary. The main point of the paper tries to show that the generated videos have clear temporal information inside a single frame. The author could raise less confusion if framing the CNN detector is a simple quantitative tool to detect the time information embedded inside the video frame.

Author response:
Thank you for addressing the concern that could potentially hinder readers from understanding our paper’s core message. The initial purpose of stating ‘CNN is inspired from humans’ was to illustrate why the problem of implicit temporal information in the generated videos is an incomprehensible phenomenon. However, as we agree that argument could be minimized and revised the paper accordingly.

Reviewer comment (Experiments):
The paper would become more appealing if framing it as a method against fake generated video and find applications in face swapping detection. Adding some experiments in this domain could make the impact broader.

Author response:
Thank you for pointing out an application of the temporal classification metric in another field. To assess the effectiveness of the proposed temporal classification metric in detecting deepfake videos, we utilized the Celeb-DF[1] and Celeb-DF-v2[1] datasets which are commonly used in the field of deepfake detection. Each dataset consists of real and synthesized videos where the synthesized videos are made by face swapping the real video with another face identity with an auto-encoder. For a fair comparison between real and synthesized videos, we first randomly select a video from real videos. Then, we select the same video with another face identity in the synthesized video. In line with our original experiments, we fix the first frame with the 0th frame from each video and randomly sample the rest 15 frames from the video. We note that the 15 frames are of the same index, meaning that they are from the same temporal location. 
The temporal classification accuracy of the synthesized video is around 1.07%p higher than real video results. As the difference is not as notable as the results in the main table (with generated and real videos), the temporal classification metric cannot be directly used for detecting deepfake videos. With curiosity we have also experimented with face-cropped versions of each video, considering that synthesized videos are generated solely by face-swapping the original video. We obtained face-cropped versions utilizing the Haar Cascade algorithm. The result followed a similar trend with the initial experiment showing around 0.92%p difference. We hypothesize that the small temporal accuracy difference comes from not generating entire videos but rather only detecting and swapping faces, thereby leaving the temporal characteristics of the real video unchanged. However, as the extended experiment on the deepfake videos may give a pathway to future research in deepfake detection, we added a section in the appendix in the revised version.

Reviewer comment (Experiments):
The authors mentioned about the human study but there is no explicit section to document the details of how to conduct the human study.

Author response:
The user study mentioned in the manuscript is to emphasize the usage of the user study in the prior works. We have not conducted any human studies for our experiment and thus do not have a section in the original paper.