Paper: Mitigating Mode Collapse in Sequential Disentanglement via an Architecture Bias (ICLR-2024)

Reviewer comment (Theory):
Technical claims put forth in this paper appear somewhat misleading in my evaluation. The authors claim that subtracting the static representation from the dynamic representation induces an architectural bias and “removes” static features from leaking into the dynamic features. It is not clear to me why this subtraction may encourage removing static information since the static and dynamic representations are not embedded into the same latent space. In fact, they are not interchangeable when input into the decoder i.e. are not separate full-dimensional latent vectors within a same latent space; rather, these representations are concatenated to form the entire image encoding. Consequently, I fail to discern any relationship between this architectural approach and a potential inductive bias.

Author response:
We would like to address your observation regarding the latent space of static and dynamic features. In our approach, both feature types are indeed encoded within the same latent space. This shared latent space is fundamental to our model’s design, allowing for the direct comparison and manipulation of static and dynamic features.
The process we describe, where static features are considered in the computation of dynamic features, is intended to explicitly separate the influence of time-invariant factors from time-variant ones. The operation to “subtract” static from dynamic features is a conceptual simplification. In practice, this is achieved through a designed inductive bias within the model architecture, ensuring that the static features do not “leak” into the dynamic representations. We fully understand that subtracting static content from dynamic factors may seem naive. By subtracting the static representation s, which is less prone to dynamic variation,  we facilitate the LSTM to concentrate solely on dynamic differences, thus streamlining its function. To empirically validate this approach, we included a series of experiments (table 4 and table 8) in the paper. These experiments specifically demonstrate that removing the subtraction operation or the loss term that penalizes the static component’s presence in dynamic factors leads to a significant drop in performance. These results substantiate the necessity and effectiveness of our subtraction method, confirming that it plays a critical role in enhancing the model’s ability to capture and separate time-varying elements accurately. Without this step, the LSTM module’s capacity to discern complete dynamic changes is notably diminished, as reflected in the lower results.

Reviewer comment (Evaluation):
Following my previous point, I do not understand why the embeddings of static and dynamic features are being plotted along each other in Fig. 2. According to my understanding, static and dynamic are components / dimensions that form concatenated latent vectors, so what is the meaning of plotting different dimensions within the same embedding space? Why should they align or be distinct? For example, each dimension can follow same gaussian distribution, but still represent different features, as the decoder can apply any function to the latent vector (which is the concatenation of static || dynamic).

Author response:
We appreciate your query regarding the visualization of static and dynamic embeddings in Figure 2. Your point touches on a critical aspect of our methodology—the representation of static and dynamic features within the same dimensional space.
In our approach, both static and dynamic features are encoded into latent vectors of the same dimensionality. This allows us to compare and contrast these features within the same embedding space, which is crucial for our analysis of disentanglement. The static features, derived from a single frame, capture the invariant aspects of the data, while the dynamic features, obtained from a sequence, encapsulate the time-varying attributes. For visualization purposes in Figure 2, we average the dynamic features across the sequence to reduce them to a single representative vector, comparable in dimension to the static features. We then employ t-SNE, to project these high-dimensional vectors into a two-dimensional space for ease of visualization and interpretation. This method allows us to observe how well the static and dynamic features are separated or entangled. The reason for plotting static and dynamic features together is to demonstrate their relationship and the effectiveness of our model in disentangling them. It is to visually confirm that the static features do not influence the dynamic representation and vice versa, which aligns with our goal of achieving clear disentanglement between invariant and variant factors. We will ensure that the revised manuscript includes a more detailed explanation of the dimensionality and handling of static and dynamic features, as well as the rationale behind their joint visualization.

Reviewer comment (Evaluation):
The quantitative evaluation includes only image-level metrics (derived from pre-trained classifiers). I believe it would be very valuable to conduct evaluation at the representation-level to measure the disentanglement quality, as the metrics proposed in [1].

Author response:
Representation-Level Evaluation and Comparison with Class-Content Disentanglement Methods:
We acknowledge the potential insights such a comparison could offer. However, our work is centered on sequential disentanglement, which inherently diverges from static disentanglement in its fundamental assumptions, modeling approach, and the applicable evaluation benchmarks. Sequential disentanglement is tailored to unravel time-related changes and maintain a consistent representation of static features over time, while class-content disentanglement typically addresses static representations where temporal dynamics do not play a role. The methodologies and tools we have developed are thus specifically designed for the challenges and nuances of sequential data. Given this significant conceptual and methodological difference, a direct comparison with class-content disentanglement methods may not adequately reflect the unique contributions and the efficacy of our approach.

Reviewer comment (Experiments):
Could the author compare their method to class-content disentanglement methods as [1]? Class could be treated as the static features and content could be referred to as the dynamic features.

Author response:
Representation-Level Evaluation and Comparison with Class-Content Disentanglement Methods:
We acknowledge the potential insights such a comparison could offer. However, our work is centered on sequential disentanglement, which inherently diverges from static disentanglement in its fundamental assumptions, modeling approach, and the applicable evaluation benchmarks. Sequential disentanglement is tailored to unravel time-related changes and maintain a consistent representation of static features over time, while class-content disentanglement typically addresses static representations where temporal dynamics do not play a role. The methodologies and tools we have developed are thus specifically designed for the challenges and nuances of sequential data. Given this significant conceptual and methodological difference, a direct comparison with class-content disentanglement methods may not adequately reflect the unique contributions and the efficacy of our approach.

Reviewer comment (Theory):
I am worried about how to ensure that s contains only static features. The authors claim that static factors can be extracted from a single frame in the sequence, which is not a necessary and sufficient condition. Otherwise, any frame from the video can be used. Why the first frame?

Author response:
- We leverage the fact that in sequential data, a single frame in a series has no time-dependent information. Therefore, a single frame will not encode dynamic features but only static features. Ablation experiments were conducted to assess the impact of utilizing different frames, and the results unequivocally demonstrated that employing the initial frame yields the most favorable outcome. We acknowledge the scenario you described, where the first data point might be more entangled, could be problematic. To mitigate this, we propose an additional check of different frames for the static factors, which we included in our paper. Our results (table 4 and table 8) show that even in cases where the first frame may not be ideal, our method still successfully disentangles the factors, as evidenced by the strong performance across different checkpoints in a sequence. This suggests that the model does not merely rely on the first frame but rather learns a robust representation of time-invariant features that are consistent across the entire sequence.

Reviewer comment (Novelty):
The method of removing static information from dynamic information is by subtraction between features, which is quite naive.

Author response:
- We understand your concern that subtracting the static part from the dynamic might sound too naive. We believe that the simplicity we bring is what makes our method unique and the empirical experiments (table 4) we performed indeed prove that this subtraction is significant to the results of the model on the benchmarks.

Reviewer comment (Theory):
How to ensure that small time-varying vectors are captured?

Author response:
- Small time-varying vectors are captured by the dynamic LSTM module, which is trained to focus on changes between frames after the static component has been subtracted. This is designed to be sensitive to even minimal dynamic variations.

Reviewer comment (Evaluation):
Does the length of the time series affect the performance of the model?

Author response:
- About your question about the length of the time series affecting the model’s performance, we have indeed tested our model on datasets with varying sequence lengths and found that our approach consistently achieves state-of-the-art performance across all of them. At the same time, since the perception of the dynamic parts in our model depends on LSTM, for particularly long series the model may fail due to the use of the LSTM network.

Reviewer comment (Evaluation):
Some of the results for example in Table 1 make the model seem way less impressive. While indeed the model surpasses all the others, the increase in performance is somewhat small. Thus I don't think being SotA is the most compelling argument for this approach.

Author response:
Thank you for pointing out the performance improvements shown in Table 1. In the paper we detail under 'Failure case analysis on MUG' why this dataset is difficult because some of the facial expressions, such as surprise and fear, look very similar. It is hard to tell them apart, even for people. Thus, little improvements are important. This means our model is really good at spotting the tiny differences in these expressions. We will take care to explain this better in our article, and see why these small steps forward are important.

Reviewer comment (Evaluation):
Similarly, there doesn't seem to be much disentanglement of the different properties in the Air Quality dataset.

Author response:
Figure 6 demonstrates the model’s ability to autonomously detect and categorize patterns within the static seasonal context of the Air Quality dataset, notably distinguishing between days with different rainfall levels. While seasonality provides a consistent background throughout the year, our model is not limited by this uniformity. It adeptly clusters days with similar weather patterns, such as grouping rainy days, which is significant as it was achieved without explicit instruction to do so. This self-guided clustering, evident in the figure, showcases the model’s nuanced comprehension of static data components. We will improve figure 6 description in our paper.

Reviewer comment (Evaluation):
Given that the authors stress the importance of mitigating mode collapse as an advantage of their model, why is there no comparison on how prevalent this issue is between their model and the alternatives? Given that as I have previously said the improvement in performance is not that great, then this would be an alternative way to support the idea that this is indeed a substantial improvement over previous approaches.

Author response:
We acknowledge its importance and in our current work, we have primarily relied on theoretical foundations and qualitative analysis to demonstrate our model's capability in mitigating mode collapse. While further empirical comparisons could be insightful, we believe the existing evidence sufficiently supports our claims. We will consider such comparisons in future research.

Reviewer comment (Experiments):
Similarly, I would like to see latent plots of alternative models as a way to compare the disentanglement properties of this approach.

Author response:
We would like to point out an advantage of our model that is relevant to this discussion: our model's flexibility in defining the dimensions of static and dynamic components. Contrasting with previous methods that allocate different dimensions for static and dynamic features, our model designates identical dimensionality to both. This consistent dimensionality is what permits us to apply t-SNE for projecting into a two-dimensional space, enabling the clear comparative visualizations showcased in our work. We would like to point out an advantage of our model that is relevant to this discussion: our model's flexibility in defining the dimensions of static and dynamic components. Other methods developed before us give different dimensions to the static component and the dynamic component. Unlike them, we gave the static factor and the dynamic factor the same dimension, which allows us to use tsnt to project the latent space into two dimensions and present it as seen in our work.