Paper: Neural Polynomial Gabor Fields for Macro Motion Analysis (ICLR-2024)

Reviewer comment (Evaluation):
Slight artifacts in boldly magnifying large motions: The paper acknowledges that when magnifying large motions, there are slight artifacts in Phase-PGF. The cause of these artifacts may stem from the neural network architecture or the way motion is represented. To reduce these artifacts, additional investigation and optimization of the model may be necessary.

Author response:
One possible reason for the slight artifacts in motion magnification is that the spatial support is insufficient for the magnified object trajectory. As we discussed in the limitation section, this might be addressed by adding more Gabor basis which requires more computational resources, or by introducing spatially-adaptive Gabor basis in future work.

Reviewer comment (Experiments):
Limited scalability to complex large-scale 3D dynamic scenes: The paper acknowledges that Phase-PGF might not perform well in complex large-scale 3D scenes due to computational efficiency issues. As the paper's focus is on macro motion analysis, addressing this issue is crucial to improving the applicability of the proposed method, especially for complex scenes. A possible solution could be the use of a spatially adaptive Gabor basis as mentioned in the paper.

Author response:
As dicussed in the limitation section, an essentialy bottleneck in scaling our method to large-scale complex 3D dynamic scene is adding more Gabor basis and subbands, which reuiqres increased compute. An alternative solution is spatially adaptive Gabor basis, yet we remain this for future work.

Reviewer comment (Experiments):
Insufficient experimental evaluation: Although the paper presents some experimental results, it would be beneficial to include more comprehensive evaluations with various datasets and tasks. For example, additional experiments could be conducted on larger and more diverse datasets, varying scene conditions, and different motion types (e.g. human motion). This would help establish the robustness and generalizability of the proposed method.

Author response:
We have included additional supporting evaluations in the revised manuscript appendix D, as detailed below:

**Additional examples.** We include 6 more examples of more complex macro motions:
- **Non-rigid motion**: We include an Internet video, *Jelly*, where a jelly is wobbling on a seat. This example demonstrates non-rigid non-uniform shearing motion.
- **Interaction between objects**: We include an example of two balls colliding with each other and then changing moving directions.
- **Multi-object motion**: We include a scene with three balls vibrating with different frequencies and magnitudes.
- **Non-periodic motion**: We synthesize three videos, *Damping*, *Bouncing*, and *Projectile*, contain different non-periodic motions. In *Damping* and *Bouncing*, we simulate a damping motion of a ball. In projectile, we simulate a ball being projected outward, subjecting to gravity.

We show further descriptions, results, and discussion in Appendix D.1. In summary, our model is able to represent and interpret the more diverse motions in these additional examples. Video results are also available at our [supplementary webpage](https://phasepgf.github.io/more_results.html).

**Quantitative results.** We also include two quantitative metrics to evaluate all the examples, including:
- Normalized Cross-Correlation (NCC) to measure for phase correlation and phase interpretability. The NCC is a standard metric in time-series analysis to measure the similarity of two 1-D temporal signals, which meet our need in measuring how the generated phase align with the motion is a dynamic scene. In Table 2, we show that our method surpasses the baseline significantly in this metric.
- Frechet Inception Distance (FID) to measure the quality of the phase-manipulated videos. In particular, we compute FID between the manipulated video and the original video. Since our motion manipulation only changes the motions presented in the scene, a lower FID then suggests less artifacts and higher visual fidelity. In Table 3, we show that the result produced by our method has better visual quality.

We include more disucssion on quantitative metrics in Appendix D.2.

**Phase space analysis and moving object discovery.** To provide more analysis, we also add an analysis on phase space. In particular, we introduce a quantity, *phase score*, to analyze and automatically select phases that correspond to significant motions in the dynamic scene. We discuss phase space in Appendix C.2. We show that our model can discover moving objects in a real video in Appendix D.6.

**Ablation study.** In Appendix D.3, we include ablation study on decoder and adversarial training. In Figure 12 of the revised manuscript, we show that the introduced components contribute significantly to our model's high visual fidelity.

**Comparison to point tracking-based method.** We include one additional point tracking-based baseline method. Specifically, we extract points using PIPs++ [1], a state-of-the-art point tracking method. To analysis motion, we use PCA to reduce its dimensions. We show a qualitative result in Figure 13. We observe that the motion space from point tracking is not structured or interpretable, and thus it does not support intrepreting or manipulating motions in a meaningful way. This suggests that the unstructured point-space representation does not easily support abstracting a low-dimensional manipulatable motion representation.

We believe these additional experiments can better support our method.

Reviewer comment (Reproducibility):
Neural network architecture: The paper uses a neural network to instantiate the Phase-PGF representation. However, the choice of the neural network architecture could be further optimized for the specific task of macro motion analysis. Including the details of the architecture may help to improve the reproducibility of the proposed work.

Author response:
In Appendix C.1, We include all our additional technical details, including
- Phase Generator 
- Spatial Polynomial Gabor Fields
- Basis Modulation and Phase-PGF
- Neural Rendering
- Feature Decoding and Rendering
- Adversarial Training

We also added Table 4,5,6,7 that further specify the hyper-parameters and network architectures we used.

Reviewer comment (Evaluation):
It would be better to design some automatic metrics other than human preference. For example, trackers can be put on moving objects to obtain ground truth motions, which can be used to compare with predicted motions. For the motion intensity adjustment experiment, to evaluate the visual quality, some metrics designed for generation tasks, e.g., FID, KID, can be used.

Author response:
Following your suggestion, we use quantitative metrics to evaluate phase interpretability and video quality. In particular, we use: 
- Normalized Cross-Correlation (NCC) to measure for phase correlation and phase interpretability. The NCC is a standard metric in time-series analysis to measure the similarity of two 1-D temporal signals, which meet our need in measuring how the generated phase align with the motion is a dynamic scene. In Table 2, we show that our method surpasses the baseline significantly in this metric.
- Frechet Inception Distance (FID) to measure the quality of the phase-manipulated videos. In particular, we compute FID between the manipulated video and the original video. Since our motion manipulation only changes the motions presented in the scene, a lower FID then suggests less artifacts and higher visual fidelity. In Table 3, we show that the result produced by our method has better visual quality.

We include more disucssion on quantitative metrics in Appendix D.2.

Reviewer comment (Experiments):
Currently, the examples in paper and supplementary website show very simple movements, mostly periodic. I wonder how would the method apply to more complex, dynamic scenes. Not necessarily dynamic 3D scenes, more complex 2D video can also do the work. If so, can the method separate more than two motions in the scene? And can some algorithm be designed to automatically find the moving object?

Author response:
We include 6 more examples of more complex macro motions:
- **Non-rigid motion**: We include an Internet video, *Jelly*, where a jelly is wobbling on a seat. This example demonstrates non-rigid non-uniform shearing motion.
- **Interaction between objects**: We include an example of two balls colliding with each other and then changing moving directions.
- **Multi-object motion**: We include a scene with three balls vibrating with different frequencies and magnitudes.
- **Non-periodic motion**: We synthesize three videos, *Damping*, *Bouncing*, and *Projectile*, contain different non-periodic motions. In *Damping* and *Bouncing*, we simulate a damping motion of a ball. In projectile, we simulate a ball being projected outward, subjecting to gravity.

We show further descriptions, results, and discussion in Appendix D.1. In summary, our model is able to represent and interpret the more diverse motions in these additional examples. Video results are also available at our [supplementary webpage](https://phasepgf.github.io/more_results.html).

### Separate more than two motions
In the Multi-object motion example above, we demonstrate separation of three balls.

### Automatic moving object discovery

To automatically discover moving objects, we introduce a quantity, *phase score*, which quantifies how much a subband contributes to the final rendering. The subbands associated with the highest phase scores represent the dynamic/static objects in the scene. We discuss the automatic phase association in Appendix C.2. We show that our model can discover moving objects in a real video in Appendix D.6.

Reviewer comment (Experiments):
Currently, the extracted motion phase is agnostic of the motion direction. Is there a way to decompose motion phases, for example, to x and y directions.

Author response:
We clarify that our method does represent motion direction by the orientation of the Gabor basis. In particular, we do decompose motion directions into $x$ and $y$. This can be seen intuitively from the projectile example in Figure 11.

Reviewer comment (Experiments):
Although the method is novel, I think the supporting experiments are insufficient to validate the method.

Author response:
Thank you for your constructive review and insightful suggestions!

We address all your concerns and questions as below.

### Additional supporting experiments
To include additional supporting experiments, we extensively include additional contents. 

**Additional examples.** We include 6 more examples of more complex macro motions:
- **Non-rigid motion**: We include an Internet video, *Jelly*, where a jelly is wobbling on a seat. This example demonstrates non-rigid non-uniform shearing motion.
- **Interaction between objects**: We include an example of two balls colliding with each other and then changing moving directions.
- **Multi-object motion**: We include a scene with three balls vibrating with different frequencies and magnitudes.
- **Non-periodic motion**: We synthesize three videos, *Damping*, *Bouncing*, and *Projectile*, contain different non-periodic motions. In *Damping* and *Bouncing*, we simulate a damping motion of a ball. In projectile, we simulate a ball being projected outward, subjecting to gravity.

We show further descriptions, results, and discussion in Appendix D.1. In summary, our model is able to represent and interpret the more diverse motions in these additional examples. Video results are also available at our [supplementary webpage](https://phasepgf.github.io/more_results.html).

**Quantitative results.** We also include two quantitative metrics to evaluate all the examples, including:
- Normalized Cross-Correlation (NCC) to measure for phase correlation and phase interpretability. The NCC is a standard metric in time-series analysis to measure the similarity of two 1-D temporal signals, which meet our need in measuring how the generated phase align with the motion is a dynamic scene. In Table 2, we show that our method surpasses the baseline significantly in this metric.
- Frechet Inception Distance (FID) to measure the quality of the phase-manipulated videos. In particular, we compute FID between the manipulated video and the original video. Since our motion manipulation only changes the motions presented in the scene, a lower FID then suggests less artifacts and higher visual fidelity. In Table 3, we show that the result produced by our method has better visual quality.

We include more disucssion on quantitative metrics in Appendix D.2.

**Phase space analysis and moving object discovery.** To provide more analysis, we add an analysis on phase space in Appendix C.2. Following the analysis on phase space, we introduce a *phase score* metric to automatically discover the associated subbands. The phase score quantifies how much a subband contributes to the final rendering. The subbands associated with the highest phase scores represent the dynamic/static components in the scene. We show that our model can discover moving objects in a real video in Appendix D.6.

**Ablation study.** In Appendix D.3, we include ablation study on decoder and adversarial training. In Figure 12 of the revised manuscript, we show that the introduced components contribute significantly to our model's high visual fidelity.

**Comparison to point tracking-based method.** We include one additional point tracking-based baseline method. Specifically, we extract points using PIPs++ [1], a state-of-the-art point tracking method. To analysis motion, we use PCA to reduce its dimensions. We show a qualitative result in Figure 13. We observe that the motion space from point tracking is not structured or interpretable, and thus it does not support intrepreting or manipulating motions in a meaningful way. This suggests that the unstructured point-space representation does not easily support abstracting a low-dimensional manipulatable motion representation.

We believe these additional experiments can better support our method.

Reviewer comment (Evaluation):
Motion editing results display visual artifact.

Author response:
### Quality for video results
As summarized above, we add a quantitative metric FID for video quality and show results in Table 8. We achieve higher quality compared to baseline methods.

Reviewer comment (Evaluation):
Sometimes it is hard to say which phase is right. Maybe participants are just biased towards more periodic-looking signal even if it does not really match the scene motion. I wonder if there is more systematic way to evaluate this? For example, we can detect the periodicity of a given motion and then compare its frequency with the predicted frequency.

Author response:
### Systematic evaluation on phases
As summarized above, we use a metric, Normalized Cross Correlation, to measure phase correctness. Please refer to appendix D.2 for more details.

Reviewer comment (Experiments):
The manipulation of the phase has been mentioned a few times, but it hasn't been detailed and demonstrated further in the paper. If the phase space is manipulatable, it will be important in the model interpretability.

Author response:
### Clarification on manipulating phases

We clarify that several of our experiments on macro motion analysis indeed manipulate phases to manipulate motions. In particular:
- **Motion separation (Figure 4)**: We used band-limit filters on the generated phase sequence, and then we isolated the motion of interest (the motion of the blue ball) according to the frequency. In this way, we manipulated (mollified) the motion of the blue ball without affecting the motion of the red ball.
- **Motion intensity adjustment (Figure 5)**: We introduced a phase augmentation technique in Section 3.2., Equation (5) and (6), to magnify the motion in the video.
- **Motion smoothing (Figure 6)**: We show that by removing high-frequency bands of the generated phase sequence, we can smooth out the high-frequency motions in the macro motion.

We also include video results of phase-based manipulation in our [supplementary webpage](https://phasepgf.github.io/more_results.html).

Reviewer comment (Reproducibility):
The implementation of using Phase-PGF in the deep neural network is not described clearly enough.

Author response:
### Details on implementation
In Appendix C.1, We include all our additional technical details, including
- Phase Generator 
- Spatial Polynomial Gabor Fields
- Basis Modulation and Phase-PGF
- Neural Rendering
- Feature Decoding and Rendering
- Adversarial Training

We also added Table 4,5,6,7 that further specify the hyper-parameters and network architectures we used.

Reviewer comment (Evaluation):
The contribution is unclear with insufficient evaluation. A more comprehensive evaluation is needed to demonstrate the boundaries of working examples, and provide more insights.

Author response:
### More extenisve quantitative evaluation
As mentioned above, we add quantitative measurements and additional examples in Appendix D of the revised manuscript. We believe these additional examples and evaluations add to the extensiveness of our experiments.

Reviewer comment (Experiments):
Quantitative evaluation: Some of the test videos are artificial. In this case, we may want to synthesize more videos that contain varying motion complexities, for a more extensive evaluation.

Author response:
### Adapting to diverse motion complexities

We include 6 more examples of more complex macro motions:
- **Non-rigid motion**: We include an Internet video, *Jelly*, where a jelly is wobbling on a seat. This example demonstrates non-rigid non-uniform shearing motion.
- **Interaction between objects**: We include an example of two balls colliding with each other and then changing moving directions.
- **Multi-object motion**: We include a scene with three balls vibrating with different frequencies and magnitudes.
- **Non-periodic motion**: We synthesize three videos, *Damping*, *Bouncing*, and *Projectile*, contain different non-periodic motions. In *Damping* and *Bouncing*, we simulate a damping motion of a ball. In projectile, we simulate a ball being projected outward, subjecting to gravity.

We show further descriptions, results, and discussion in Appendix D.1. In summary, our model is able to represent and interpret the more diverse motions in these additional examples. Video results are also available at our [supplementary webpage](https://phasepgf.github.io/more_results.html).

Reviewer comment (Experiments):
Ablation study: An ablation study for each network module would be beneficial.

Author response:
### Ablation study

In Appendix D.3, we include ablation study on decoder and adversarial training. In Figure 12 of the revised manuscript, we show that the introduced components contribute significantly to our model's high visual fidelity.

Reviewer comment (Evaluation):
Evaluation metrics: A metric to measure if the phase meets the requirements would be useful.

Author response:
### Evaluation metrics

To allow automatic quantitative measures, we additionally include the following evaluation metrics:
- Normalized Cross-Correlation (NCC) to measure for phase correlation and phase interpretability. The NCC is a standard metric in time-series analysis to measure the similarity of two 1-D temporal signals, which meet our need in measuring how the generated phase align with the motion is a dynamic scene. In Table 2, we show that our method surpasses the baseline significantly in this metric.
- Frechet Inception Distance (FID) to measure the quality of the phase-manipulated videos. In particular, we compute FID between the manipulated video and the original video. Since our motion manipulation only changes the motions presented in the scene, a lower FID then suggests less artifacts and higher visual fidelity. In Table 3, we show that the result produced by our method has better visual quality.

We include more disucssion on quantitative metrics in Appendix D.2.

Reviewer comment (Experiments):
Some existing methods use different formulations to perform similar motion analysis, and it is strongly recommended to add the comparison with them. For example, some point tracking methods can also extract rigid body movements as sparse points, and using these predictions it's also possible to extract interpretable features of the motion, such as phase or other high-level descriptions. While the paper employs a *unique* methodology, it would be beneficial to add more experiments and discussions on these approaches.

Author response:
### Comparison to point tracking
We follow your suggestion to compare with a point tracking-based method. In particular, we extract points using PIPs++ [1], a state-of-the-art point tracking method. To analyze motion, we use PCA to reduce its dimensions. We show a qualitative result in Figure 13. We observe that the motion space from point tracking is not structured nor interpretable, and thus it does not support intrepreting or manipulating motions in a meaningful way. This suggests that the unstructured point-space representation does not easily support abstracting a low-dimensional manipulatable motion representation.


[1] Zheng et al., PointOdyssey: A Large-Scale Synthetic Dataset for Long-Term Point Tracking, ICCV, 2023

Reviewer comment (Experiments):
Where is the boundary of the phase space this method can support? There can be highly varied frequencies in the real world, but the working range of the proposed method is unclear. It would be useful to explore these boundaries by conducting experiments with various artificial videos.

Author response:
### The working range of the proposed method
Besides the five additional examples we have shown above, we also add failure cases to demonstrate the working range. In particular, we find that our approach does not work with motions of shapeless objects such as fire (Figure 15), because it violates our assumption on time-invariant basis functions to represent the video appearances. Another example is chaotic unpredictable motions. We show a hair example in Figure 15 where the hair motion is unpredictable and it might not allow a frequency-based low-dimensional representation.

Reviewer comment (Experiments):
How to determine the number of phases? In some cases, there appears to be one phase, while in others there are multiple. Is it possible to use an excessive number of phases to overfit a scene? Conversely, what would be the outcome if we used a single phase for a scene with multiple objects?

Author response:
### How to initialize the number of phases

We add a detailed discussion in Appendix C.2 of the revised manuscript. The main idea is that, $K$ can be set either according to prior knowledge of how many motion components there are in the scene, or to a large number. When we set a large $K$, our model can automatically associate appropriate subbands (i.e., phases) to the motion components in the scene, thus automatically discovering moving objects without manually specifying the number of phases. Then, all other redundent phases have little or no effects on representing the dynamic contents. 

To do this, we introduce a *phase score* metric to automatically discover the associated subbands. The phase score quantifies how much a subband contributes to the final rendering. The subbands associated with the highest phase scores represent the dynamic objects in the scene. We show that our model can discover moving objects in a real video in Appendix D.6.

Reviewer comment (Evaluation):
The extracted phases seem to care about the frequency more compared to the amplitude variations. In the second video of Fig.2, different balls do have different spatial transitions, but they seem to share the same peak.

Author response:
### Amplitude of phases
We clarify that the motion amplitudes can be captured by the generated phase amplitudes within **a single motion**. This can be observed from the *Damping* example in Figure 7.

Yet, we note that the phase amplitudes are not comparable among different motions. For a scene with multiple motions, such as the two bouncing balls, although each ball's motion amplitude can still be captured, the relative phase amplitudes of the two balls do not reflect their relative motion amplitudes, because each phase signal is generated from an implicit function (phase generator), and an implicit function can have an arbitrary output scale.

Reviewer comment (Novelty):
There is an option to use point tracking to extract the phase from the tracking path. How does this method complement them, and in what ways does it offer unique advantages compared to other methods?

Author response:
### Comparison to point tracking
We follow your suggestion to compare with a point tracking-based method. In particular, we extract points using PIPs++ [1], a state-of-the-art point tracking method. To analyze motion, we use PCA to reduce its dimensions. We show a qualitative result in Figure 13. We observe that the motion space from point tracking is not structured nor interpretable, and thus it does not support intrepreting or manipulating motions in a meaningful way. This suggests that the unstructured point-space representation does not easily support abstracting a low-dimensional manipulatable motion representation.


[1] Zheng et al., PointOdyssey: A Large-Scale Synthetic Dataset for Long-Term Point Tracking, ICCV, 2023

Reviewer comment (Experiments):
Are the extracted phases open to manipulation? For example, if we multiply the phase by a coefficient A, will the corresponding recovered image or video be amplified? It is claimed in the paper that we can manipulate, but the results seem to be missing.

Author response:
### Clarification on manipulating phases

We clarify that several of our experiments on macro motion analysis indeed manipulate phases to manipulate motions. In particular:
- **Motion separation (Figure 4)**: We used band-limit filters on the generated phase sequence, and then we isolated the motion of interest (the motion of the blue ball) according to the frequency. In this way, we manipulated (mollified) the motion of the blue ball without affecting the motion of the red ball.
- **Motion intensity adjustment (Figure 5)**: We introduced a phase augmentation technique in Section 3.2., Equation (5) and (6), to magnify the motion in the video.
- **Motion smoothing (Figure 6)**: We show that by removing high-frequency bands of the generated phase sequence, we can smooth out the high-frequency motions in the macro motion.

We also include video results of phase-based manipulation in our [supplementary webpage](https://phasepgf.github.io/more_results.html).