Paper: FLD: Fourier Latent Dynamics for Structured Motion Representation and Learning (ICLR-2024)

Reviewer comment (Experiments):
Limited Data Set: The paper heavily relies on a specific dataset for evaluation, which might not be representative of various motion patterns and scenarios. Hence, the proposed method's performance might not be generalizable to other datasets or real-world applications.

Author response:
We thank the reviewer for raising concerns about the representativeness of our dataset and the generalizability of GLD. We would like to highlight a few features that our dataset possesses.
- Dataset Diversity: Our dataset encompasses a wide range of human locomotion types, providing a diverse foundation for evaluation. This includes varied motions such as jogging, running, and striding, ensuring a comprehensive motion spectrum is covered​​.
- Generalization Capability: GLD's ability to generalize is demonstrated through its successful reconstruction and prediction of unseen motions, such as the diagonal run scenario in Section 5.2. This showcases its effectiveness beyond the specific motions present in the training dataset​​.
- Adaptability in Real-time Tracking: The system's adaptability is further evidenced in Section 5.3 by its proficiency in real-time tracking of diverse user inputs, including motions like 'spinkick,' which are significantly different from the training data​​.
- Handling Motion Transitions: GLD effectively handles transitions between different motion types, as shown in our experiments with interpolated movements in Section 5.3, highlighting its capacity to maintain coherence in motion sequences​​.


Additionally, we would like to draw attention to the fact that the GLD training does not assume specific embodiments of input sequences. In fact, any periodic or quasi-periodic signal can be effectively encoded by GLD and converted to quasi-constant frequency domain representations. These signals, or reference motions, are not necessarily physically compatible and thus not necessarily achievable by a specific embodied agent. Therefore, the performance of the controller depends on policy learning, rather than data diversity.

Reviewer comment (Experiments):
Controller's Adaptability: Although the motion learning controller is designed to adapt its tracking strategy dynamically, the paper lacks a thorough analysis of the controller's adaptability in handling various motion patterns and unseen targets (for example, multiple intersecting targets). Further study could help establish the controller's robustness and versatility.

Author response:
In response to the reviewer's comment on the controller's adaptability, we acknowledge the importance of this aspect and direct attention to Section 5.3 of our paper, which details experiments showcasing the controller's adaptability across various motion patterns and transitions. For a more dynamic demonstration, we invite reviewers to view videos on our [project website](https://sites.google.com/view/iclr2024-gld/home), which include scenarios of the controller responding to multiple and unseen motion inputs. 

We acknowledge the need for more challenging input to test the limit of the controller's performance. However, as a design parameter, we would also like to highlight the influence of the fallback threshold $\epsilon_{GLD}$ in the control robustness and versatility. The design of this threshold determines the conservativeness of the tracking. In the extreme case, the fallback mechanism is either always triggered and thus the controller rejects tracking any input target, or the fallback mechanism is always off and thus the controller attempts to track every target motion. We conducted additional experiments on these cases and presented the result in a video titled “Fallback Ablation” on our [project website](https://sites.google.com/view/iclr2024-gld/home).

Reviewer comment (Experiments):
Limited Adaptability to Other Domains: The paper discusses motion representation and learning in robotics. However, it may not be easily transferable to other domains, such as human motion analysis, due to differences in motion characteristics.

Author response:
We acknowledge that our current work primarily focuses on motion representation and learning in physics-based character and robotic control. However, it's essential to highlight the inherent versatility of the GLD framework. GLD is fundamentally designed to encode any periodic or quasi-periodic signal, transforming these inputs into quasi-constant frequency domain representations. This capability is not restricted to robotic embodiments or specific motion learning tasks.

The core strength of GLD lies in its ability to provide a concise and meaningful representation of complex, high-dimensional, long-horizon, non-linear signals. While our demonstrations and experiments are rooted in robotics, the underlying principles of GLD make it a valuable tool for a broader range of applications, including human motion analysis. The architecture and approach of GLD do not inherently limit it to robotic systems; rather, its potential applications can extend to any field requiring efficient and accurate representation of periodic or quasi-periodic signals.

Looking forward, we are optimistic about the adaptation and application of GLD in various other domains. We believe that the framework's ability to distill complex motion dynamics into manageable representations will find relevance and utility in fields such as biomechanics, sports science, and even animation, where understanding and replicating human or character motions are crucial.

Reviewer comment (Experiments):
Data Quality: The paper fails to address the possible negative effects of poor-quality data on the proposed GLD method's performance in real-world applications. Poor-quality data could potentially degrade the effectiveness and accuracy of the GLD method in real-world applications.

Author response:
In response to the reviewer's concern regarding the impact of poor-quality data, our research has indeed taken into account the variable quality of real-world data. As detailed in Section 5, the human locomotion clips used in our study captured in [1] include elements like noise and aperiodicity. GLD addresses these issues by enforcing a quasi-constant parameterization, which effectively enhances the sequence quality through periodic reconstructions.

The fundamental design of GLD is to encode periodic or quasi-periodic signals into constant frequency domain representations, a process that is resilient to data quality variances. This ensures that GLD's effectiveness remains robust, regardless of the imperfections in the input data. It's important to note that while the quality of the data may not affect the efficacy of GLD itself, it does have implications for how downstream tasks, such as motion learning, might utilize these data.

Further addressing the issue of data quality, we specifically explore scenarios in Section A.2.11 of the appendix where certain components of the motion data are unlearnable. Our findings, illustrated in Figure S15, show that through adaptive curriculum learning methods, the learning agent can shift its focus towards targets with more potential for improvement, avoiding areas with limited prospects. This adaptability in handling varied data quality is a testament to GLD's utility and effectiveness in practical, real-world applications.

Reviewer comment (Theory):
Can GLD be extended to handle non-periodic motions?

Author response:
Potentially yes. Our framework assumes quasi-constant parameterization for motion sequences that are periodic or quasi-periodic in nature, which are common in human movements. 

In cases where motion sequences do not exhibit periodic characteristics, applying a globally constant set of latent parameterizations can lead to underparameterization of the latent embeddings. This, in turn, may result in less accurate reconstructions of the original motion sequences. We have thoroughly addressed this limitation in Section A.3.1 in the appendix of our paper, with an exemplified non-periodic trajectory. For a more detailed exploration of this issue, we refer you to both Section A.3.1 and Figure S16 in our paper. 

However, we can propose a potential approach for accommodating such motions within the GLD framework. Essentially, any non-periodic motion can be conceptualized as a periodic sequence, where the period is equivalent to the length of the entire motion. By adopting this perspective, GLD can effectively encode non-periodic motions, provided that the trajectory segment window $H$ is sufficiently extended to encompass the full length of the motion. This adaptation allows GLD to process and represent non-periodic motions by essentially treating them as extended periodic sequences, thereby leveraging its inherent strengths in encoding and analyzing motion data.

Reviewer comment (Experiments):
How does GLD perform in long-term learning tasks, and can it adapt to and learn new tasks in open environments?

Author response:
Yes. GLD's adaptability in long-term learning tasks and open environments is demonstrated through its adaptive curriculum learning approach, as detailed in Sections A.2.10 and A.2.11 of our paper's appendix. By initially training on an offline reference dataset, GLD effectively navigates and expands the motion space during training, generating and adapting to new targets. This is facilitated by an online fitted Gaussian Mixture Model sampling strategy based on Absolute Learning Progress (ALPGMM, [2]), which directs the system's focus towards promising and underexplored motion regions, as shown in Figure S15.

The method's adaptability is further evidenced by its expansion into previously unexplored motion areas, while maintaining expert performance on mastered motions. This guided exploration significantly increases the motion coverage, demonstrating GLD's ability to adapt to and learn in dynamic and open-ended scenarios. We would direct the reviewer to Sections A.2.10 and A.2.11 for detailed discussion.

Reviewer comment (Evaluation):
How can the stability and safety of GLD be ensured when applied in real-world scenarios?

Author response:
The fallback mechanism in GLD introduced in Section 4.3.2 is an integral safety feature designed to manage situations where the motion learning controller encounters inputs that are significantly different from its training data. These scenarios potentially lead to poor performance or unsafe actions. The primary function of this mechanism is to intuitively filter out risky motion inputs, providing an essential safeguard against unpredictable behavior that could arise from unanticipated motion inputs. 

We have conducted an additional experiment to demonstrate the utility of the fallback mechanism. This experiment specifically showcases scenarios where the mechanism is disabled, highlighting the contrast in the system's behavior and performance. To provide a clear and accessible illustration of these differences, we have included a video titled “Fallback Ablation” on our [project website](https://sites.google.com/view/iclr2024-gld/home). This video visually demonstrates the potential risks and performance issues that arise when the fallback mechanism is not employed, thereby underlining its effectiveness and importance.

Reviewer comment (Experiments):
How does the GLD perform with noisy motion trajectories? It would be important to investigate how GLD performs in real-world scenarios where motion data may be corrupted or incomplete.

Author response:
We would like to direct the reviewer to our previous response to **Weakness 4** in light of GLD’s performance in dealing with poor-quality motion trajectories.

Reviewer comment (Theory):
Can GLD be extended to multi-agent motion learning? How does the computation complexity increase as the number of targets increases?

Author response:
GLD is primarily a self-supervised representation learning framework, focusing on the efficient encoding of motion trajectories. It does not inherently encompass agent learning aspects. The application of GLD in the context of motion learning is currently integrated with a reinforcement learning controller that is designed for single-agent scenarios. Extending this framework to multi-agent settings is an intriguing concept; however, it falls outside the scope of our current research. As the number of agents increases, the complexity of the learning environment and the interactions between agents would likely lead to a rise in computational demands. This could involve more sophisticated state representation and decision-making processes to effectively handle the dynamics of multi-agent interactions.

In summary, while GLD's current implementation is not directly geared toward multi-agent learning, we recognize the importance of this area for future research. Exploring the applicability and scalability in multi-agent settings and its computational implications presents a valuable direction for subsequent work in this field.

Reviewer comment (Reproducibility):
The authors should make better effort to make the paper self-contained in the main manuscript, and not rely excessively on the supplementary material. There is a severe lack of details in the main manuscript, due to the authors moving them to the supplementary material.

Author response:
Thanks for pointing this out. To address the reviewer’s concerns, we modified Section 4.3.1 to give an example of the skill samplers and refer to the appendix for more variants. We also added the notation of the uniform sampling range $\mathcal{U}$ accordingly.

In our initial submission, Sec 5.4, together with Suppl. A.2.10 and A.2.11 primarily served as an in-depth exploration of skill sampler design in motion learning. This extended study, though not affecting the GLD pipeline and conclusion, reinforces our argument for the versatility and applicability of the GLD latent parameterization space.

However, as suggested by the reviewer, we realized that presenting this abundant but auxiliary exploration in the appendix may be intensive and distracting. Therefore, we adjusted Sec 5.4 accordingly as an extended discussion to direct interested readers to ablation studies in the appendix while keeping the core innovations of our work in the main text. We also restructured the appendix by removing redundant experiments, discussions and figures to further improve readability. We hope the modified structure could highlight our focus and main contribution in the main text.

Reviewer comment (Presentation):
One form of the skill sampler should be included in the main section.

Author response:
Thanks for pointing this out. To address the reviewer’s concerns, we modified Section 4.3.1 to give an example of the skill samplers and refer to the appendix for more variants. We also added the notation of the uniform sampling range $\mathcal{U}$ accordingly.

In our initial submission, Sec 5.4, together with Suppl. A.2.10 and A.2.11 primarily served as an in-depth exploration of skill sampler design in motion learning. This extended study, though not affecting the GLD pipeline and conclusion, reinforces our argument for the versatility and applicability of the GLD latent parameterization space.

However, as suggested by the reviewer, we realized that presenting this abundant but auxiliary exploration in the appendix may be intensive and distracting. Therefore, we adjusted Sec 5.4 accordingly as an extended discussion to direct interested readers to ablation studies in the appendix while keeping the core innovations of our work in the main text. We also restructured the appendix by removing redundant experiments, discussions and figures to further improve readability. We hope the modified structure could highlight our focus and main contribution in the main text.

Reviewer comment (Writing):
$\mathcal{U}$ in Fig.2 is unclear from the main section.

Author response:
Thanks for pointing this out. To address the reviewer’s concerns, we modified Section 4.3.1 to give an example of the skill samplers and refer to the appendix for more variants. We also added the notation of the uniform sampling range $\mathcal{U}$ accordingly.

In our initial submission, Sec 5.4, together with Suppl. A.2.10 and A.2.11 primarily served as an in-depth exploration of skill sampler design in motion learning. This extended study, though not affecting the GLD pipeline and conclusion, reinforces our argument for the versatility and applicability of the GLD latent parameterization space.

However, as suggested by the reviewer, we realized that presenting this abundant but auxiliary exploration in the appendix may be intensive and distracting. Therefore, we adjusted Sec 5.4 accordingly as an extended discussion to direct interested readers to ablation studies in the appendix while keeping the core innovations of our work in the main text. We also restructured the appendix by removing redundant experiments, discussions and figures to further improve readability. We hope the modified structure could highlight our focus and main contribution in the main text.

Reviewer comment (Presentation):
Section 5.4 seems unnecessary, as all the content is in the supplementary material.

Author response:
Thanks for pointing this out. To address the reviewer’s concerns, we modified Section 4.3.1 to give an example of the skill samplers and refer to the appendix for more variants. We also added the notation of the uniform sampling range $\mathcal{U}$ accordingly.

In our initial submission, Sec 5.4, together with Suppl. A.2.10 and A.2.11 primarily served as an in-depth exploration of skill sampler design in motion learning. This extended study, though not affecting the GLD pipeline and conclusion, reinforces our argument for the versatility and applicability of the GLD latent parameterization space.

However, as suggested by the reviewer, we realized that presenting this abundant but auxiliary exploration in the appendix may be intensive and distracting. Therefore, we adjusted Sec 5.4 accordingly as an extended discussion to direct interested readers to ablation studies in the appendix while keeping the core innovations of our work in the main text. We also restructured the appendix by removing redundant experiments, discussions and figures to further improve readability. We hope the modified structure could highlight our focus and main contribution in the main text.

Reviewer comment (Presentation):
The figures seem unorganized, as the readers are asked to refer to figures in a random order, including the supplementary material. The colored lines in Figures 5 and 6 is unclear. There should be some sort of explanation of each element in all the figures.

Author response:
We thank the reviewer for the constructive feedback regarding the references to appendix figures in the main text. We restructured the figure reference and believe the changes will significantly improve the readability and coherence of our paper.

Thanks for the feedback on the clarity of the colored lines in Figures 5 and 6. In these figures, the colored curves represent different states corresponding to various quantities, such as linear and angular velocities ($v$ and $\omega$), and arm and leg joint positions ($q_{arm}$ and $q_{leg}$). Each curve illustrates the evolution of these quantities over time, aligned with the focus of our paper on periodic motion dynamics.

We acknowledge that a more detailed explanation of each element within the figures would enhance their clarity. However, we faced the challenge of balancing the depth of explanation with the constraints of space and the need to maintain a focus on the core aspects of our research. Our priority was given to present the periodic nature and the dynamics of the motion trajectories, which we felt were central to the paper’s primary concern. However, we are happy to include the color information as suggested in the final version, provided that more space is available in the main text.

Reviewer comment (Evaluation):
The effectiveness of the fallback mechanism must be discussed more quantitatively. The authors only discuss the results in Section 5.3, but there is no concrete evidence that the mechanism worked well. There should be some statistics regarding if the motion prediction actually failed when the fallback mechanism was not introduced. Such objective evidence is lacking for the readers to decide whether it is effective or not.

Author response:
Thanks for the feedback regarding the need for more concrete evidence supporting the effectiveness of the fallback mechanism. The fallback mechanism in GLD is an integral safety feature designed to manage situations where the motion learning controller encounters inputs that are significantly different from its training data. These scenarios potentially lead to poor performance or unsafe actions. The primary function of this mechanism is to intuitively filter out risky motion inputs, providing an essential safeguard against unpredictable behavior that could arise from unanticipated motion inputs. 

In response to the request for more concrete evidence, we have conducted an additional experiment to demonstrate the utility of the fallback mechanism. This experiment specifically showcases scenarios where the mechanism is disabled, highlighting the contrast in the system's behavior and performance. To provide a clear and accessible illustration of these differences, we have included a video titled “Fallback Ablation” on our [project website](https://sites.google.com/view/iclr2024-gld/home). This video visually demonstrates the potential risks and performance issues that arise when the fallback mechanism is not employed, thereby underlining its effectiveness and importance.

The effectiveness of the fallback mechanism is thus straightforward. Its role in maintaining safety and reliability in motion learning is evident from the clear difference in system behavior observed when the mechanism is disabled. We hope this evidence presented in our materials offers a direct and comprehensible demonstration of the mechanism’s role in enhancing system safety and robustness.

Reviewer comment (Theory):
Was there any drawback from introducing the assumption that some elements are generally constant throughout a sequence? Were there actions that did not present these characteristics?

Author response:
Yes. This assumption is indeed a pivotal aspect of our method, and we have carefully considered its potential limitations. As correctly noted, our framework assumes quasi-constant parameterization for motion sequences that are periodic or quasi-periodic in nature, which are common in human movements. However, we acknowledge that this approach may not be ideally suited for non-periodic trajectories.

In cases where motion sequences do not exhibit periodic characteristics, applying a globally constant set of latent parameterizations can lead to underparameterization of the latent embeddings. This, in turn, may result in less accurate reconstructions of the original motion sequences. We have thoroughly addressed this limitation in Section A.3.1 in the appendix of our paper, with an exemplified non-periodic trajectory. For a more detailed exploration of this issue, we refer you to both Section A.3.1 and Figure S16 in our paper. 

However, we can propose a potential approach for accommodating such motions within the GLD framework. Essentially, any non-periodic motion can be conceptualized as a periodic sequence, where the period is equivalent to the length of the entire motion. By adopting this perspective, GLD can effectively encode non-periodic motions, provided that the trajectory segment window $H$ is sufficiently extended to encompass the full length of the motion. This adaptation allows GLD to process and represent non-periodic motions by essentially treating them as extended periodic sequences, thereby leveraging its inherent strengths in encoding and analyzing motion data.

Reviewer comment (Presentation):
The action classes indicated by the colors in Fig.5 seems to be missing, making the evaluation of these latent spaces difficult. What do each color respond to?

Author response:
The color assignment of the latent offset in Fig 5 (right) was indicated in the caption in our initial submission. Red corresponds to step in place, yellow corresponds to forward run, and green corresponds to forward stride. To improve the readability as suggested by the reviewer, we added a legend at the top. We hope this helps.

Reviewer comment (Writing):
The motion learning/control policy part of this work needs a clearer problem formulation. It is unclear what the exact goal is here, and a lot is left to the readers to infer. The reviewer gathers that the goal is to learn to generate a series of motions that track a desired motion sequence, using the motion prediction, but it is unclear why control parameters need to be generated to do so, instead of just encoding the target motions directly. A clearer problem description will help avoid such confusion.

Author response:
We appreciate the reviewer’s feedback on the motion learning formulation. The primary objective of this second phase is to develop an RL control policy that enables a **physics-based** character or robot to effectively track a series of desired motion sequences generated using the predictive capabilities of our GLD model. In the context of physics-based motion control, direct assignment of state values to the agent is not feasible. Unlike kinematic models where states can be explicitly set, physics-based models include system dynamics and thus must adhere to the laws of physics and environmental interactions. 

Specifically, our control policy outputs desired joint positions $q*$ for the robot, which are further converted to actuator torques and then applied to control the robot’s movements within the simulation environment (detailed in Section A.2.1 in the appendix). This process ensures that the robot’s movements adhere to physical constraints and realistically respond to environmental interactions.

To clarify this aspect, we have revised Section 4.3 of the manuscript to include a more detailed explanation of the problem space in physics-based motion learning to avoid further ambiguity.

Reviewer comment (Presentation):
Much of the interesting work around control policies is in the appendices.

Author response:
Thanks for pointing this out. In our initial submission, Sec 5.4, together with Suppl. A.2.10 and A.2.11 primarily served as an in-depth exploration of skill sampler design in motion learning. This extended study, though not affecting the GLD pipeline and conclusion, reinforces our argument for the versatility and applicability of the GLD latent parameterization space.

However, as suggested by the reviewer, we realized that presenting this abundant but auxiliary exploration in the appendix may be intensive and distracting. Therefore, we adjusted Sec 5.4 accordingly as an extended discussion to direct interested readers to ablation studies in the appendix while keeping the core innovations of our work in the main text. We also restructured the appendix by removing redundant experiments, discussions and figures to further improve readability. We hope the modified structure could highlight our focus and main contribution in the main text.

Reviewer comment (Novelty):
This work appears closely related to a phase functioned neural network described in Holden et al., "Phase-Functioned Neural Networks for Character Control", which computes network weights using a cyclic function controlled by phase. The proposed approach uses an autoencoder and has clear differences (PFNN does next state prediction), but the core idea is similar and PFNN also considers aspects like motion blending. The paper is mentioned in passing in the introduction, but a discussion on this in the related work section is recommended given the similarity in the core idea.

Author response:
We appreciate the reviewer's suggestion for pointing us to the PFNN work, with which our work on GLD shares several key similarities. In particular, as in PFNN, our method predicts future motion and factorizes a phase vector from the high-level motion features, which are further represented in the network weights in PFNN and in a latent space $\Theta$ in GLD.

In addition to the motion prediction feature that both methods share, we would like to highlight that the crux of our contribution lies in the development of a latent parameterization space, $\Theta$, which would be difficult with PFNN. This latent parameterization space serves a dual purpose in our framework: firstly, it provides a concise yet comprehensive representation of motions, and secondly, it enables meaningful interpolation and generalization of these motions. This aspect is particularly crucial for training effective tracking policies that can adapt to a wide range of target motions, a feature not directly addressed by PFNN.

PFNN, in its methodology, primarily aims at directly predicting future states from past states and user inputs, thereby bypassing the need for an intermediary motion representation. In contrast, our work emphasizes constructing a robust and versatile motion representation space from which a **tracking policy** can dynamically select and adapt to various motion trajectories. This difference in focus and application underlines the unique nature of our contribution, which we felt was more aligned with the structured motion representation methods discussed in our related works section.

Reviewer comment (Presentation):
The paper is extensive, with a number of detailed appendices. While this is good, references to appendix figures in the main text body are rather distracting.

Author response:
We thank the reviewer for the constructive feedback regarding the references to appendix figures in the main text. We restructured our manuscript and believe the changes will significantly improve the readability and coherence of our paper.

Reviewer comment (Presentation):
Assumption 1 – It should be pointed out earlier that since the latent space is learned, this can be enforced.

Author response:
We thank the reviewer for pointing this out. The text under Assumption 1 is modified accordingly.

Reviewer comment (Presentation):
Figure 5 is confusing; it is assumed that the right y-axis is an error metric. The axis and caption should be labeled to indicate these measures.

Author response:
Yes, the label caption ($e$) was a bit off in our initial submission. We updated the figure accordingly. Thanks for pointing it out.