Paper: Interpretable Latent Distributions Using Space-Filling Curves (ICLR-2024)

Reviewer comment (Novelty):
Lack of Novelty and Originality: The paper primarily focuses on the application of an existing method to new models and datasets, without introducing any novel or original contributions. There is no apparent effort to adapt or modify the method to suit the characteristics of the new models and dataset. In my view, this lack of novelty and originality raises questions about whether this paper is suitable for submission to ICLR, as it could be perceived as a dual submission.

Author response:
1. Thanks for your invaluable comments! The SFVQ presented in  (Vali and Bäckström, 2023) was originally used for universal interpretation of the underlying morphological structure of the latent space (or any distribution). However, in this paper apart from those universal interpretations, we discovered that in a SFVQ's curve the line connecting two corner points can work as an interpretable direction for changing attributes. Also we noticed that SFVQ works similar to PCA as its lines lies along the directions in which the data has the most variance (Fig. 5), but its main difference with PCA is that SFVQ's lines are not orthogonal to each other. According to this property, we can state why  in Figure 7 our "Gender" direction remains more in the valid range than PCA-based GANSpace, in which "Gender" and "Rotation" are orthogonal directions to each other.
In addition, we showed that SFVQ can be considered as a sorted VQ method which can perform better than traveling salesman problem in sorting especially for sparse data (Section 5 and Figure 9). According to section 5, we also showed that we can use the huge number of data points lied on the SFVQ’s curve as meaningful points for controllable data augmentation (Fig 13 and Fig 14 in Appendix). These are all new things we found on top of the original SFVQ paper. Furthermore, according to Figure 2, we discovered 13 different directions for FFHQ dataset, which is more than the number of directions found in most of the state-of-the-art papers. Also, some directions are really specific and unique to our proposed method such as “Asian Face”, “Hat”, “Graduation Hat”, “Beard” for human faces (FFHQ), “Age” for animals (AFHQ).

Reviewer comment (Evaluation):
Insufficient Experimental Rigor: The experimental setup in the paper is notably lacking in terms of rigor. The absence of quantitative metrics and reliance solely on visualizations is a significant shortcoming. Without comprehensive metrics, the results can be susceptible to cherry-picking and fail to provide a strong empirical validation of the method.

Author response:
2. About the quantitative metric, thanks to the recommendation from one of the reviewers, I am going to apply the quantitative measures used in [1] (commutativity error, side effect error, and identity error) and compare our discovered directions with some existing state-of-the-art methods. Unfortunately, I think I cannot complete the experimentation before the rebuttal period ends as it takes some time.

[1] Aoshima, Takehiro, and Takashi Matsubara. "Deep Curvilinear Editing: Commutative and Nonlinear Image Manipulation for Pretrained Deep Generative Model." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.

Reviewer comment (Experiments):
The paper does not discuss or compare with two very relevant recent works [1, 2]. These works are both unsupervised and model-agnostic, and, similarly to GANSpace, should necessarily be included in the experimental evaluation of the proposed method.

Author response:
1. Thanks for your invaluable comments! We read these papers and it sounds interesting and we are excited to measure the performance of our interpretable directions regarding the quantitative measures in [1] (commutativity error, side effect error, and identity error) and compare it with other works. However, it seems that the implementation of [1] is not provided. So, we will do the comparisons with [2] and maybe other existing works. Unfortunately, I think I cannot complete the experimentation before the rebuttal period ends as it takes some time.

Reviewer comment (Evaluation):
The presented empirical evaluation is only qualitative, rendering impossible to compare its advantages against GANSpace (and other relevant works that are missing [1, 2]). Even in that case, in the only figure that provides comparison with an existing work (i.e., Fig. 7, comparison with GANSpace), it is hardly visible why the proposed method is better.

Author response:
2. About the comparison with GANSpace (Fig. 7), those 5 directions are the only directions provided by GANSpace for StyleGAN2 on FFHQ dataset that we could use for qualitative comparisons. About the performance comparison, among the first three directions (Gender, Rotation, Smile) which are the most prevalent directions in the papers in the literature, we can easily say that regarding Gender our direction is more valid or has higher validity length than GANSpace. Also regarding Smile our direction make more sensible changes.

Reviewer comment (Evaluation):
In absence of any quantitative comparison, it is impossible to assess the merits of the proposed method. Appropriate quantitative metrics can be found in [1, 2]. In my point of view, the paper should compare with those methods using the suggested metrics.

Author response:
1. Thanks for your invaluable comments! We read these papers and it sounds interesting and we are excited to measure the performance of our interpretable directions regarding the quantitative measures in [1] (commutativity error, side effect error, and identity error) and compare it with other works. However, it seems that the implementation of [1] is not provided. So, we will do the comparisons with [2] and maybe other existing works. Unfortunately, I think I cannot complete the experimentation before the rebuttal period ends as it takes some time.

Reviewer comment (Experiments):
The provided results for ImageNet (BigGAN) are limited. I could only find two examples limited to the Dog category. It would be interesting to see more visualizations of how the method works on other categories, especially objects.

Author response:
Thanks for your invaluable comments! We will also find some useful directions for other BigGAN classes (like an object) and update the paper with the new directions for sure.

Reviewer comment (Experiments):
The author has not discussed and evaluated how the latent-space inversion could be performed in the proposed latent space, and whether the discovered directions are meaningful and effective for the inverted latent code.

Author response:
We were aware of some methods which experimented their directions in GAN inversions. However, we have not experimented the performance of our discovered directions in GAN inversions, since the performance would be dependent on the accuracy and efficiency of the inversion model as well. We will consider this case and might experiment the performance of our method in GAN inversions.

Reviewer comment (Experiments):
I could not find an ablation on the number of the codebook vectors. I would appreciate it if the authors could further clarify this.

Author response:
If we apply the SFVQ with 2 bits, we can only consider 3 different directions (as there are 3 lines connecting the 4 corner points). Hence, the more the bitrate is, the more diverse and unique directions we could find by plotting and observing generations from corner points. With lower bitrates (like 2 bits in figure 5) the learned SFVQ finds the most fundamental directions in which the data has the most diversity along these directions in the latent space like Rotation and Gender. Therefore, the directions in figure 2 are found by observing the learned SFVQ (only once) over different bitrates ranging from 2 to 12 bits.

Reviewer comment (Evaluation):
I would assume sampling only on the obtained curve could affect the image quality and diversity quite a lot. Currently, no evaluations (e.g. FID scores) are provided for the generation quality.

Author response:
As the main objective of the paper was improving the interpretablity of the latent space, we did not study the SFVQ’s generation quality. However, according to section 5, the SFVQ’s curve lies mainly inside the data distribution. Hence, if the we sample points from the SFVQ’s curve, the corresponding generated images are meaningful with good quality. Regarding the diversity, if we sample the points existed on the line between two corner points, we would not get diverse generations, as the points exhibits the transition from image A to image B, which are the corner point images. However, if we add noise to the sampled points from SFVQ’s curve (figures 13 and 14 in the Appendix) we could get more diverse and meaningful generations.