Paper: Centroid- and Orientation-aware Feature Learning (ICLR-2024)

Reviewer comment (Presentation):
In general, the presentation could be significantly improved. It seems to me that the key contribution of the method is the introduction of loss L_m but the discussion does not make this clear.

Author response:
1. **Improvement of our paper (W1)**

* We apologise for the omission of this proof since translational equivariance in CNN is a well-known concept in the community, and the proof is similar to the Theorem 3.2. We have now added the proof of theorem 3.1 in the Appendix. 

* Related to ‘training and inference performance’: We apologise for the confusion. It refers to the training and inference time. We have changed it in the paper.

* Related to L_m loss: We highlight that 
> Although image moments capture the centroid and orientation features with a high degree of accuracy, they can still have a degree of subtle inaccuracy. Therefore, we utilize the image moments as guiding principles during the initial stages of learning.

Reviewer comment (Writing):
It is suggested (end of section 2.3) that the “primary focus” of the paper might be to achieve disentanglement (and indeed, some experiments also suggest that). However, it is not clear how this is pursued besides obtaining moments and orientation.

Author response:
2. **Problem statement and motivation of our paper (W2)**

Thank you for the comments. We apologize if the motivation seems disconnected from the experimental validation. We have changed the first paragraph of the introduction to clarify our problem as follows: 

‘The distinct 2D projections that arise from the rotation of a 3D object can be seen as unique categories of in-plane rotation. Each category corresponds to a specific orientation in the 3D space. This principle is widely applied in fields such as computer graphics, computer vision, and related areas where the understanding and manipulation of 3D objects in a 2D images are essential. In this context, robust learning of centroids and orientations of objects in images, along with other features, is important, especially when objects are captured with 3D shifts and rotations such as protein, and galaxy studies.’

Therefore, we demonstrate that the capability of the model to learn key features, including centroids and orientations in the experimental validation. Table 3 and 4 show that DAE and beta-VAE have poor performance in factorizing two datasets, XYRCS and dSprites. On the other hand, the disentanglement scores of the proposed model on these datasets demonstrate that the model effectively learn all features including centroids and orientations.

Reviewer comment (Presentation):
The theorems in section 3.1 are barely referenced in subsequent sections.

Author response:
3. **Optimal latent dimension for TARGET-VAE and IRL-INR (W3, Q2)**

Target-VAE and IRL-INR conducted their experiments with large latent dimension, pursuing two objectives: 1) learning centroids and orientation, and 2) reconstructing the original data. Therefore, the two methods do not address the disentanglement scores in their paper, while they included the centroids and the orientation correlation. Therefore, the latent dimensions chosen in their respective papers are optimal for reconstructing the original data but may not effectively factorize all features. In contrast, our goal is not only to learn centroids and orientation but also to independently learn the other features while successfully reconstructing the original data. To show this, we chose a smaller latent dimension because modularity (or independence) and compactness are important in disentanglement properties. Figure 2, 4, 5, and 7 along with Table 3 and 4 demonstrate the capability of the model to factorize all the features while reconstructing the original data.

Reviewer comment (Evaluation):
The motivation seems disconnected from the experimental validation. It is stated that learning of centroids and orientations “underpins” a number of downstream tasks. It is not clear what this means nor is it clear what the level of success of the proposed approach would be in this regard.

Author response:
2. **Problem statement and motivation of our paper (W2)**

Thank you for the comments. We apologize if the motivation seems disconnected from the experimental validation. We have changed the first paragraph of the introduction to clarify our problem as follows: 

‘The distinct 2D projections that arise from the rotation of a 3D object can be seen as unique categories of in-plane rotation. Each category corresponds to a specific orientation in the 3D space. This principle is widely applied in fields such as computer graphics, computer vision, and related areas where the understanding and manipulation of 3D objects in a 2D images are essential. In this context, robust learning of centroids and orientations of objects in images, along with other features, is important, especially when objects are captured with 3D shifts and rotations such as protein, and galaxy studies.’

Therefore, we demonstrate that the capability of the model to learn key features, including centroids and orientations in the experimental validation. Table 3 and 4 show that DAE and beta-VAE have poor performance in factorizing two datasets, XYRCS and dSprites. On the other hand, the disentanglement scores of the proposed model on these datasets demonstrate that the model effectively learn all features including centroids and orientations.

Reviewer comment (Experiments):
It is unclear what tables 3 and 4 convey when comparing different models as the optimal latent dimension is model dependent. For instance, for models TARGET-VAE and IRL-INR the original authors showed results for d >= 32 (but it is suggested d = 2 in the experiments in the present submission).

Author response:
3. **Optimal latent dimension for TARGET-VAE and IRL-INR (W3, Q2)**

Target-VAE and IRL-INR conducted their experiments with large latent dimension, pursuing two objectives: 1) learning centroids and orientation, and 2) reconstructing the original data. Therefore, the two methods do not address the disentanglement scores in their paper, while they included the centroids and the orientation correlation. Therefore, the latent dimensions chosen in their respective papers are optimal for reconstructing the original data but may not effectively factorize all features. In contrast, our goal is not only to learn centroids and orientation but also to independently learn the other features while successfully reconstructing the original data. To show this, we chose a smaller latent dimension because modularity (or independence) and compactness are important in disentanglement properties. Figure 2, 4, 5, and 7 along with Table 3 and 4 demonstrate the capability of the model to factorize all the features while reconstructing the original data.

Reviewer comment (Writing):
Is the method pursuing disentanglement of all features or mainly/only obtaining moments and orientation? In case of the former, this is not clear in the presentation — how is this pursued?

Author response:
4. **Objective of our paper (Q1)**

We pursue disentanglement of all features. DAE already shows an ability to disentangle features. However, it fails factorizing all features when the data has a rotational feature. Therefore, we introduce a new architecture outlined in the paper with $L_{moment}$ to explicitly learn centroids and rotations, leading to fully factorized features.

Reviewer comment (Experiments):
Why are baselines not compared with representation dimension as in the original papers?

Author response:
3. **Optimal latent dimension for TARGET-VAE and IRL-INR (W3, Q2)**

Target-VAE and IRL-INR conducted their experiments with large latent dimension, pursuing two objectives: 1) learning centroids and orientation, and 2) reconstructing the original data. Therefore, the two methods do not address the disentanglement scores in their paper, while they included the centroids and the orientation correlation. Therefore, the latent dimensions chosen in their respective papers are optimal for reconstructing the original data but may not effectively factorize all features. In contrast, our goal is not only to learn centroids and orientation but also to independently learn the other features while successfully reconstructing the original data. To show this, we chose a smaller latent dimension because modularity (or independence) and compactness are important in disentanglement properties. Figure 2, 4, 5, and 7 along with Table 3 and 4 demonstrate the capability of the model to factorize all the features while reconstructing the original data.

Reviewer comment (Experiments):
Due to the deterministic and simple nature of the moment computation, it could be easy for the neural network to learn z_eq. Therefore, a result on ablating L_{moment} could be interesting to see the emergent properties just based on the reconstruction loss, and also could be a baseline, since moment loss is the only new component here. As a corollary, the moment loss could also be applied over other baselines to evaluate how much does it contribute in improving their performance.

Author response:
1. **Ablation study (W)**

We conduct the ablation study, and the results indicate that the disentanglement scores increase with the inclusion of the $L_{moment}$. We will include the ablation study table in the paper and post it separately in the response soon. 

---

1. **Ablation study**

Following our earlier response, our contribution is not limited on $L_{moment}$. We have also Introduced a translational and rotational equivariant encoder. We have conducted the ablation study on the XYRCS and dSprites datasets without $L_{moment}$ or the translational and rotational equivariant encoder (TREE). The results show that both $L_{moment}$ and TREE are essential to achieve improved disentanglement scores. CODAE is a combination of DAE, TREE, and $L_{moment}$.

* **XYRCS dataset**
| Features | Models | z-diff | z-var | irs | dci | sap | mig | jemmig | dcimig | avg |
|---|---|---|---|---|---|---|---|---|---|---|
| XYRCS | **CODAE** | **0.97** | **0.76** | 0.67 | **0.83** | 0.61 | **0.67** | **0.72** | **0.63** | **0.73** |
|  | DAE+TREE| 0.82 | 0.73 | 0.59 | 0.48 | 0.37 | 0.22 | 0.48 | 0.24 | 0.49 |
|  | DAE+$L_{moment}$ | 0.86 | 0.74 | **0.87** | 0.57 | **0.63** | 0.44 | 0.61 | 0.51 | 0.65 |
| CS | **CODAE** | **1.00** | **1.00** | **0.82** | **0.99** | **0.83** | **0.90** | **0.77** | **0.90** | **0.90** |
|  | DAE+TREE| 0.92 | 0.81 | 0.38 | 0.35 | 0.13 | 0.16 | 0.47 | 0.16 | 0.42 |
|  | DAE+$L_{moment}$ | 0.57 | 0.78 | 0.45 | 0.20 | 0.00 | 0.01 | 0.29 | 0.01 | 0.28 |




* **dSprites dataset**
| Features | Models | z-diff | z-var | irs | dci | sap | mig | jemmig | dcimig | avg |
|---|---|---|---|---|---|---|---|---|---|---|
| XYRCS | **CODAE** | **0.97** | **0.85** | **0.82** | **0.56** | **0.59** | **0.48** | **0.64** | **0.45** | **0.67** |
|  | DAE+TREE| 0.92 | 0.78 | 0.65 | 0.36 | 0.34 | 0.30 | 0.45 | 0.21 | 0.50 |
|  | DAE+$L_{moment}$ | 0.71 | 0.56 | 0.42 | 0.24 | 0.33 | 0.19 | 0.37 | 0.14 | 0.37 |
| SS | **CODAE** | **1.00** | **0.99** | **0.74** | **0.89** | **0.51** | **0.58** | **0.71** | **0.64** | **0.75** |
|  | DAE+TREE| 0.95 | 0.83 | 0.29 | 0.61 | 0.38 | 0.55 | 0.61 | 0.53 | 0.59 |
|  | DAE+$L_{moment}$ | 0.84 | 0.53 | 0.34 | 0.33 | 0.44 | 0.40 | 0.46 | 0.38 | 0.46 |

Reviewer comment (Novelty):
Novelty of the moment loss is very limited, as it is a widely known concept in the community.

Author response:
2. **Contributions of our paper (W)**

While the concept of moment loss is widely recognized in the community, this paper introduces the first method to incorporate image moments into the learning mechanism -  which is not common. In addition, our contribution is not limited on $L_{moment}$. We:
* Introduce a translational and rotational equivariant encoder.
* Propose an architecture that eliminates computationally expensive decoder, such as spatial or hypernetwork decoders.
* Drastically reduce computational costs while achieving superior disentanglement scores.

Reviewer comment (Evaluation):
GF-Score could be reported as proposed in the DAE paper.

Author response:
4. **GF-Score (W)**

Thank you for the comments. We have added the scores in the paper and will post it separately in the response soon. 

---

2. **GF-Score**

We have posted the GF-Score of two real datasets, namely EMPIAR-10029 (proteins) and Galaxy Zoo, that do not have labels. As specified in [1], a lower value is preferred. Additionally, it's important to note that the EMPIAR-10029 dataset comprises  distinct 2D projections resulting from the rotation of a 3D object.

| GF-Score |  |  |
|---|---|---|
| Models/Datasets | EMPIAR-10029 | Galaxy-zoo |
| CODAE | **0.000039** | **0.000088** |
| Spatial-VAE | 0.000820 | 0.004332 |
| Target-VAE | 0.000766 | 0.002456 |
| IRL-INR | 0.000995 | 0.001182 |

[1] Orthogonality-Enforced Latent Space in Autoencoders: An Approach to Learning Disentangled Representations, ICML2023