Paper: Beyond Disentanglement: On the Orthogonality of Learned Representations (ICLR-2024)

Reviewer comment (Experiments):
only 2 datasets and models (as compared to e.g., Locatello et al. 2019) are compared in Section 4.3. Please consider using all the 7 datasets from *disentanglement_lib,* otherwise the choice seems a bit arbitrary

Author response:
- **W1: Expansion of Dataset Usage**
    - **Inclusion of SmallNorb Dataset**: In response to your suggestion, we have incorporated the SmallNorb dataset into our study. This dataset offers a significant variance from Dsprites and Cars3D, providing a more comprehensive evaluation of our metric.
   - **Planned Inclusion of DSprites Variants**: For the camera-ready version of our paper, we plan to introduce three further variations of the DSprites dataset available from disentanglement_lib. This addition will further broaden the scope of our evaluation and align with the diversity presented in Locatello et al. 2019. Note that Shapes3D is unfortunately not readily available from disentanglement_lib.

Reviewer comment (Presentation):
Figure 1 is quite difficult to grasp. I understand that the concept is not trivial to present (and the caption is already lengthy), but maybe you could consider extending/rewriting the caption to make it clearer? Perhaps in a step wise manner (multiple figures). I find it crucial for conveying the idea of your paper. If you lack space I believe figure 3 could be compressed/removed instead

Author response:
- **W2: Clarity of Figure 1**
    - **Revision of Figure 1**: Following the feedback from multiple reviewers, we have replaced the original Figure 1 with a more straightforward representation in the main text, moving the original figure to the appendix. This change aims to simplify the initial understanding of our concept.
    - **Enhanced Caption and Explanation**: As you suggested, we have elaborated on the figure's caption and provided a more detailed step-wise explanation.

Reviewer comment (Experiments):
The sub-optimal performance of Explicitness for the perfectly disentangled case could stem from overfitting. How do the authors handle this problem with their metric? What were the train/test splits used for the experiments? How sensitive is the metric to smaller sample sizes?

Author response:
**Responses to Specific Questions:**

- **Q1 (Handling of Overfitting)**: To mitigate overfitting, we have employed a validation set, separate from our test set, for early stopping during model training. We have added this and further training details in *Appendix B: Experimental Details*
- **Q2 (Train/Test Splits)**: For our experiments, we reserved 15% of each dataset for testing purposes. We then used the test set for calculating the importances for the IWO metric, ensuring that our evaluations are based on unseen data. Note that the subspace basis are part of the model and learned during training. 
- **Q3 (Metric Sensitivity to Sample Sizes)**: Recognizing the importance of this concern, we have added a dedicated experiment in *Appendix B.4: IWO on Limited Data*. We investigates the sensitivity of our metric to smaller sample sizes, providing valuable insights into its robustness and reliability in varied data conditions.

Reviewer comment (Experiments):
My main concern is about the insufficiency of evaluation. Give that $\beta$-TCVAE was a few years ago and there have been a large number of variants of VAEs that do disentangled representations, I would hope that the authors can implement a few more models for comparison.

Author response:
- **Expanded Model and Dataset Evaluation:**
Addition of SmallNORB and Further Models: In direct response to the reviews, we have expanded our evaluation to include the SmallNORB dataset. This addition, along with the inclusion of two more models - Factor VAE and $\beta$-VAE - enriches our analysis and addresses the diversity of scenarios you highlighted. These new elements complement our existing use of Dsprites, Cars3D, Annealed VAE, and $\beta$-TCVAE. In the camera-ready version of the paper, we plan to introduce three further variations of the DSprites dataset provided by disentanglement_lib.
- **Rationale for Dataset and Model Selection:**
    - **Consistency with** ***disentanglement_lib***: Our decision to use the Dsprites, Cars3D, SmallNORB datasets, and the selected models is grounded in their presence within *disentanglement_lib* https://github.com/google-research/disentanglement_lib Locatello et al., 2019 [1]. This library is a benchmark in disentanglement research and offers unparalleled reproducibility and comparability. Utilizing these datasets and models minimizes biases that could arise from custom implementations or training procedures, thus providing a clear and objective evaluation of our proposed metric's efficacy.
    - **Advantages of Established Benchmarks:**  Employing resources from *disentanglement_lib* ensures that our study aligns with prevailing research standards, enhancing the reliability and validity of our findings. This approach also allows our results to be directly comparable with a broad spectrum of existing studies, which is crucial for contextualizing and validating our contributions.

Reviewer comment (Experiments):
There are also very commonly used datasets that were not considered here, e.g. CelebA, Shape3D, Clevr, etc. I would like to see results on these more complex data.

Author response:
- **Expanded Model and Dataset Evaluation:**
Addition of SmallNORB and Further Models: In direct response to the reviews, we have expanded our evaluation to include the SmallNORB dataset. This addition, along with the inclusion of two more models - Factor VAE and $\beta$-VAE - enriches our analysis and addresses the diversity of scenarios you highlighted. These new elements complement our existing use of Dsprites, Cars3D, Annealed VAE, and $\beta$-TCVAE. In the camera-ready version of the paper, we plan to introduce three further variations of the DSprites dataset provided by disentanglement_lib.
- **Rationale for Dataset and Model Selection:**
    - **Consistency with** ***disentanglement_lib***: Our decision to use the Dsprites, Cars3D, SmallNORB datasets, and the selected models is grounded in their presence within *disentanglement_lib* https://github.com/google-research/disentanglement_lib Locatello et al., 2019 [1]. This library is a benchmark in disentanglement research and offers unparalleled reproducibility and comparability. Utilizing these datasets and models minimizes biases that could arise from custom implementations or training procedures, thus providing a clear and objective evaluation of our proposed metric's efficacy.
    - **Advantages of Established Benchmarks:**  Employing resources from *disentanglement_lib* ensures that our study aligns with prevailing research standards, enhancing the reliability and validity of our findings. This approach also allows our results to be directly comparable with a broad spectrum of existing studies, which is crucial for contextualizing and validating our contributions.

Reviewer comment (Theory):
I wonder why the $\Delta$ L can be used to measure the importance. Could you justify it in more detail?

Author response:
- **Q1 (Justification of $\Delta \mathcal{L}$ as an Importance Measure)**: $\Delta \mathcal{L}_l$ measures how much the expected loss of regressing a generative factor increases when regressing from a subspace with dimension $l$ vs. $l-1$. It is therefore directly linked to the dimension which lies in the null-space of the projection from $l$ to $l-1$. As such, we use $\Delta \mathcal{L}_l$ to allocate the importance which that particular dimension (which is lost in the projection) plays for the generative factor in question.
 In light of your feedback, we have refined the explanation in our manuscript. Additionally we have adjusted the importance-weighting mechanism in IWO to better reflect the variation in overall model informativeness, following the insights from Eastwood et al., 2018 [1].

Reviewer comment (Theory):
Is the reason that you choose to only apply linear projection using $W_{1:L}$ is technical difficulty or indeed conceptual purpose?

Author response:
- **Q2 (Use of Linear Projections)**: Our choice to use linear projections was driven by both conceptual and technical considerations. Conceptually, linear projections are a natural fit for assessing orthogonality in latent spaces. Technically, their efficient implementation via Linear Neural Networks (LNNs) further motivated our choice.

Reviewer comment (Reproducibility):
While presentation quality is generally adequate, the paper misses to convey all the necessary details to facilitate reproduction of the method and of the study. This lack of technical detail in the main body is not compensated by the availability of appendices, supplementary materials or code. One key aspect that is unclear to me is how one is expected to identify the generative factors set $z_1,\dots, z_K$ and how such $K$ is determined in general. The method involves training a potentially large amount of regressors and little information is provided on how this is done in practice (e.g. how much should the training be pushed in terms of regression error? What are the stopping conditions? How are the linear model initialised?).

Author response:
- **W1: Enhancing Reproducibility and Methodological Clarity**
    - **Appendix and Implementation Details:** We have created a detailed appendix providing comprehensive information about implementation and model choices. This includes specifics on the initialization of linear layers, network training procedures, and hyper-parameter optimization strategies.
    - **Code Repository**: To further facilitate reproducibility, we have established an anonymous code repository that contains all necessary code and documentation: https://anonymous.4open.science/r/iwo-E0C6/README.md

Reviewer comment (Novelty):
The positioning with respect to the literature is on the weak end. The paper misses to discussion and cite works formalising weaker forms disentanglement [A, B, C]. In particular, it would seem relevant to discuss the relationship between the proposed approach and those building on (and measuring) linear symmetry-based disentanglement [B,D].

Author response:
- **W2: Expanding the Literature Review**
    - **Weaker forms of Disentanglement**: We have integrated a thorough discussion on linear symmetry-based disentanglement into our related work section. This includes an analysis of the works you mentioned and others in the field, emphasizing the relationship between our approach and these methods. We have also included a segment on causal representation learning, particularly focusing on how it intersects with and differs from disentanglement methodologies, including ours.

Reviewer comment (Experiments):
While the experiments are generally well-designed, the evidence they provide does not seem enough to support the major claims of this paper. As long as one departs from the ideal setting, it is difficult to assess the added value of IWO over DCI and MIG. Additional experiments are needed on more challenging datasets, such as ModelNet40 and COIL-100, possibly enlarging the scope of methods to compare with by including those in [B,D]. It would also be of help to qualitatively explore the impact of the proposed methodology, e.g. by exploring the effects of manipulating the representations over the relevant subspaces “suggested” by the metrics.

Author response:
- **W3: Strengthening the Experimental Section**
    - **SmallNORB Inclusion**: We agree with your point on the need for more challenging datasets. While ModelNet40 and COIL-100 are indeed valuable, we have chosen to incorporate the SmallNORB dataset beside DSprites and Cars3D. It offers comparable complexity, with diverse variations in object types, camera angles, and lighting conditions, aligning closely with the complexity of ModelNet40 and the natural imagery of COIL-100, while being part of *disentanglement_lib* https://github.com/google-research/disentanglement_lib/tree/master by Locatello et al., 2019 [1] which offers unparalleled reproducibility and comparability. Utilizing datasets and models from *disentanglement_lib* also minimizes biases that could arise from custom implementations or training procedures, thus providing a clear and objective evaluation of our proposed metric's efficacy. In the camera-ready version of the paper, we plan to introduce three further variations of the DSprites dataset provided by disentanglement_lib.
    - **Additional Downstream Task Experiment:** We have added an experiment involving a downstream task which demonstrates the interplay between subspace structures and IWO, offering a deeper insight into the practical implications of our methodology. See *Appendix B.3: Modulation Task.*

Reviewer comment (Experiments):
The proposed methodology seems very computationally involved. I am using the word “seems” as the paper lacks a comparative assessment of the cost of the method. This should be done while considering more realistically sized problems, involving latent spaces of non-trivial size.

Author response:
- **W4: Addressing Computational Costs**
    - **Comparative Assessment**: We acknowledge the importance of understanding the computational cost of our method. We have added a section that showcases the computational requirements of our approach for 60 runs on the Small Norbs dataset. We have also added the necessary strategies to deal with high dimensional latent spaces, such as larger reductions between consecutive LNN heads and an initial LNN-layer of smaller size than the latent space.  See *Appendix C: Computational Resources Analysis*.

Reviewer comment (Novelty):
Can the Authors please discuss the relationship with linear symmetry-based metrics?

Author response:
- **Q2 (Linear Symmetry-Based Metrics Relationship)**: In our expanded literature review, we discuss the relationship between our approach and linear symmetry-based metrics, delineating both the commonalities and distinctions.

Reviewer comment (Experiments):
The empirical analysis would be substantially strengthened by adding new experiments on as ModelNet40 and COIL-100, considering also computational costs?

Author response:
- **Q3 (Empirical Analysis Enhancement)**: We have addressed this in W3, as discussed above. The addition of the SmallNORB dataset, additional models, an analysis on reduced data (*Appendix B.4: IWO on limited data*) and the new downstream task experiment significantly bolster our empirical analysis.