Paper: Structural Adversarial Objectives For Self-Supervised Representation Learning (ICLR-2024)

Reviewer comment (Theory):
The motivation behind the proposed method is not sufficiently clear for me. Despite the authors providing an ablation study, the principles behind the different losses are not well explained. I expect the authors to provide a more reasonable motivation to help readers understand the necessity of the proposed method beyond experimental results.

Author response:
Our proposed loss functions seek to synchronize the distributions of real and generated images, as in Equation 2. Additionally, we aim to automate learning of semantic representations. This is achieved through the structuring of non-collapsed representations by optimizing their covariance (Eqn.2) and by creating local clusters through feature grouping (Eqn.6). Figure 2 illustrates the effectiveness of our proposed objectives, showing that by optimizing the discriminator with these objectives, the learned embedding aligns semantically similar data and separates dissimilar data.

Reviewer comment (Novelty):
The paper is the lack of discussion and comparison with the relevant work, ContraD [1], which splits the discriminator into feature learning and real/fake discrimination, similar to the motivation of the work.

Author response:
We appreciate the reference to the ContraD study and plan to include it in our related work section. ContraD enhances representation learning in GAN frameworks by incorporating view-consistency contrastive learning methods. Their goal differs remarkably from our approach, which demonstrates that adversarial training objectives can inherently encapsulate feature learning capabilities. Our method presents a concise learning paradigm that does not rely on any form of view-consistency objectives, whether explicit or implicit.

Reviewer comment (Experiments):
The generation performance of the proposed method is unsatisfactory, according to the FID results in Table 4. While there is an improvement compared to the outdated BigGAN, it is not an appropriate baseline for current comparison. Since the authors have compared their proposed method to StyleGAN2-ADA, to substantiate their claim of improved image generation quality, it would be beneficial for them to compare it to StyleGAN2-ADA on the same architecture.

Author response:
In our work, the primary focus is on representation learning rather than image generation. This is why we used a ResNet-18 as the discriminator in all experiments, ensuring consistent comparison with other representation learning frameworks. Although the choice of generator is secondary, it is crucial that it does not become a limiting factor in our pipeline. We opted for BigGAN's generator with increased feature channels for this reason. We did experiment with StyleGAN's generator but found it less effective for our purposes.

Reviewer comment (Theory):
Why did the authors choose to implement JSD as the loss function? Could a distance metric like Wasserstein-2 distance, commonly used in FID, also based on the assumption of Gaussian distributions, can be used?

Author response:
The JSD was chosen for its properties of symmetry, bounded values and allowing for the simultaneous optimization of mean and variance. Although the Wasserstein-2 distance similarly optimizes both mean and variance, its unbounded nature can lead to greater instability compared to JSD, making JSD a more stable and suitable choice for our objectives.

Reviewer comment (Experiments):
Given that the loss function involves the computation of covariance and Jacobian matrices, which can be computationally expensive, could the authors provide a comparison of training time and overheads with the baselines?

Author response:
We have benchmarked the running times of each section in the discriminator's (D's) round, with the times reported as average values over 100 runs. The measurements were taken using a batch size of 256 and a single A40 GPU. The time taken for each section is as follows:

- Forward pass of the discriminator: 0.02s

- Forward pass of the generator: 0.07s

- Computing the model's approximated Jacobian: 0.27s

- Computing the adversarial loss: 0.007s

Running time for a single forward-backward pass is:

- A Complete D's round with our objectives: 0.78s

- A Complete D's round with vanilla GAN objectives: 0.29s

Our proposed objectives require about 2.7x the time of the GAN baseline during training, a reasonable time budget given the significant improvement in feature learning ability. It should be noted that, during inference time, there is no additional running time overhead compared to the GAN baseline.

Reviewer comment (Experiments):
Can the authors conduct parameter analysis experiments to provide guidance on the selection of hyperparameters?

Author response:
We run ablation experiments by sweeping through the choices of $\lambda_h,\lambda_c, \lambda_s$ in CIFAR-10 and show the results as follows:

|$\lambda_c$|$\lambda_s$|$\lambda_h$| Linear-SVM|
|:---:|:---:|:---:|:---:|
|3 | 5 | 2 | 89.1|
|3 | 5 | 4 | 89.8|
|2 | 5 | 4 | 88.7|
|1 | 5 | 4 | 88.6|
|1 | 3 | 4 | 88.7|

**Table C: Ablation experiments on hyper-parameters.** Our method is insensitive to the hyperparameter change within a reasonable range.

Reviewer comment (Evaluation):
It is also not clear why the authors use SVM and K-M for evaluating the learned representations in Table 1 and do not include linear probing, which is commonly used in the representation literature.

Author response:
**A2**: We utilized Linear-SVM for evaluation, applying the default hyperparameters provided by SKlearn. This approach, compared to linear probing that uses mini-batch gradient descent, minimizes randomness by running over the entire dataset in a single batch. As Linear-SVM is also a linear classifier, we anticipate both classifiers reaching similar end states, a claim supported by the almost identical results in Table 2 for both Linear-SVM and Linear probing. We employed K-Means clustering to evaluate the features under more challenging conditions than linear classifiers, offering a unique perspective on the quality of the learned features.

Reviewer comment (Experiments):
The reconstruction-based self-supervised methods (e.g., MAE), which have been shown to outperform contrastive learning methods on ImageNet-1k, also do not rely on hand-crafted data augmentations. Hence, to demonstrate the contribution of this work, it is necessary to show that the proposed method can provide performance gain over them on large-scale datasets.

Author response:
**A3**: Masked Autoencoders (MAEs) implictly employ the view-consistency objectives through a process of implicit matching between corrupted and clean images. On the other hand, they are sensitive to hyperparameters like patch size and mask ratio. In linear probing evaluations, MAEs underperform compared to contrastive learning methods. This is evidenced in MAGE [1], where the MAE-VIT-B achieves a 68$\%$ probing accuracy, in contrast to the 76.7$\%$ achieved by the MoCo v3.

Reviewer comment (Novelty):
I think the authors missed a very relevant related work (not my paper) which should be discussed and compared with: Li et al. MAGE: MAsked Generative Encoder to Unify Representation Learning and Image Synthesis. CVPR 2023.

Author response:
**A4**: We thank reviewer for the suggestion of MAGE and we plan to include it in our related work section. MAGE enhances representation learning through masked autoencoders by adding a contrastive learning method. This differs significantly from our approach, which aims to demonstrate a straightforward learning paradigm free from any form of view-consistency objectives, whether explicit or implicit.

Reviewer comment (Novelty):
The cluster property is well-known in the discriminator. Since DCGAN already show it, so I think it is not new in this paper to present it.

Author response:
While the discriminator's feature learning capability in traditional GAN frameworks, like DCGAN, is acknowledged, our work focuses on enhancing this aspect significantly. As evidenced in Table 4, conventional GANs exhibit limited feature learning ability. Our innovations in training objectives and regularization techniques have remarkably improved feature learning performance, as shown by the increase in K-Means clustering scores from 29.69$\%$ to 80.11$\%$, and Linear classifier scores from 69.31$\%$ to 89.76$\%.

Reviewer comment (Experiments):
The used datasets are small. I would like to use big datasets to support the proposed method.

Author response:
Please see the large-scale experiments section in our overall reply.

Reviewer comment (Experiments):
Also the frameworks are out of fashion. I think the well-known architecture (e.g., stylegan) is more convincing.

Author response:
Our primary goal is to advance representation learning, which is why we used a ResNet-18 as the discriminator across all experiments for consistent comparison with other representation learning frameworks. Although the choice of generator is secondary, it is crucial that it does not become a limiting factor in our pipeline. We opted for BigGAN's generator with increased feature channels for this reason. We did experiment with StyleGAN's generator but found it less effective for our purposes.

Reviewer comment (Presentation):
There are not much visualization results.

Author response:
We have included various visualizations in our paper, such as the learned embedding (Fig 1(b)), the dynamic updates of the learned embedding in a synthetic scenario (Fig 2), and generated images in the Appendix.

Reviewer comment (Writing):
The aims of the paper are not consistently clear throughout. From the introduction: "...also improves the quality of samples produced by the generator." From Section 3.1: "their motivation is to improve image generation quality rather than learn semantic features — entirely different from our aim." This inconsistency somewhat weakens the contribution of the paper.

Author response:
Our work primarily focuses on representation learning, not on enhancing generation quality. Improved image quality emerges as a secondary benefit from our proposed objectives. These goals are not necessarily in conflict, though their relative importance might vary depending on application. Unlike ICGAN, which uses an external feature learner to improve their generation quality, our framework is designed to learn representations from adversarial training. We refer to ICGAN in our paper to highlight the fundamental differences between their approach and ours.

Reviewer comment (Experiments):
The biggest issue is the lack of the major comparison dataset in vision representation learning: full ImageNet. This omission is surprising for several reasons: (1) It is commonly used in existing literature. (2) BigGAN, whose architecture inspires the proposed work, is trained on ImageNet. (3) The compared methods are significantly hampered with such small data.

Author response:
Please see the large-scale experiments section in our overall reply.

Reviewer comment (Experiments):
In the comparison to the Masked Auto Encoder (MAE) paper, the reduced training dataset size of ImageNet10, as well as the smaller patch size, is a fairly large deviation. Furthermore, there is no mention of what representation space is used from the MAE: all image patches or the CLS token? While these are fine details, they are crucial for fair comparison. Given the issues with MAE, there is concern that similar problems may exist with other compared methods as well.

Author response:
We provide implementation details for the methods we compared, including MAE, in Appendix A.2. For MAE, we used the average of all patch tokens from the encoder's output as the representation for the linear classifier.

Reviewer comment (Writing):
The use of z and z^g is a little confusing, as z usually refers to the generator’s input and z^g even more so.

Author response:
We use $z$ to denote the encoder's final representation, and $v$ as the input for the generator. This is in contrast to the traditional GAN framework, which only has a single latent space as the generator's input. Our model includes an additional latent space, which is the encoder's output.

Reviewer comment (Writing):
The fine-grained clustering description is confusing. Clarification is needed on how the memory bank works. Specifically, is z^b the discriminator representation of the real images encoded into the latent space? The nomenclature is not clear. A plain English explanation of what the loss function is accomplishing would be illuminating.

Author response:
The memory bank holds the encoder's representations of real images. It is implemented as a first-in-first-out queue, updated continuously by replacing the oldest features with new ones from the current mini-batch.