Paper: ADDP: Learning General Representations for Image Recognition and Generation with Alternating Denoising Diffusion Process (ICLR-2024)

Reviewer comment (Theory):
$z_{t-1}$ and $\bar{z}_{t-1}$ are conditionally independent, but why does using $z_t$ to predict $\bar{z}_{t-1}$ result in significantly better performance?

Author response:
There may be a misunderstanding on the experiment setting. Specifically, according to Eq.(6) in the paper, if we change $z\_t$ to $z\_{t-1}$ in the condition, the input image will be generated from $z\_{t-1}$ and $\bar{z}\_{t-1}$ for the consistency of timestep. As a result, the input inadvertently incorporates the essential information of the training targets. Therefore, the main reason for the poor generation performance stems from the leakage of information when using $z\_{t-1}$ to predict $\bar{z}\_{t-1}$.

To avoid confusion, we have revised our paper to remove the statement of conditionally independent.

Reviewer comment (Evaluation):
Due to the encoder being trained on noisy images, there is a significant drop in performance in Linear Probing.

Author response:
First, it is important to emphasize that the core capability of self-supervised learning (SSL) lies in fine-tuning, not linear-probing. Poor linear-probing should not negate the models. MAE [2] and BEiT [3] have highlighted this point. Still, we recognize that this is a potential direction for future research, and we will take this aspect into consideration in future work.

Nevertheless, we explored the reasons for our poor linear-probing. As discussed in Sec. 4.2, our model is trained on noisy synthetic images, making it hard in linear-probing on real images. We speculate that training on real images may help improve the linear-probing performance. To validate this, we fine-tuned our final model for 50 epochs with a 50% input mask ratio and z0 as targets. The linear-probing result improved from 11.5% to 41.2% (better than BEiT), indicating that using data closer to real images indeed enhances linear-probing performance.

Reviewer comment (Experiments):
Does expanding the training dataset to include both original images and noisy images improve image recognition performance?

Author response:
Thanks for your suggestion. We tried to mix original images and noisy images during training with a mixing ratio of 1:1. We list the results below. It shows that incorporating original images can improve image recognition performance. Additionally, we observe that incorporating original images will also cause a decline in the quality of image generation. This may be due to the inconsistency of image distribution between training and inference, and the mixing ratio of 1:1 may not be appropriate. The optimal mixing ratio may need further study to achieve a balance between recognition and generation performances.
| Training Data | FID $\downarrow$ | IS $\uparrow$ | FT Top1 acc $\uparrow$ |
| --- | --- | --- | --- |
| noisy images | 23.6 | 33.1 | 83.5 |
| original:noisy=1:1 | 214.6 | 3.1 | 83.6 |

Reviewer comment (Presentation):
The illustrated masked tokens $\bar{z}_{t-1}$ and $z_{t-1}^{pred}$ in Figure 4 are not matching.

Author response:
Thanks for your careful reading. We have fixed it in the revised version.

Reviewer comment (Writing):
The text is hard to follow. Additional copy editing may help here.

Author response:
A1: We apologize for the inconvenience caused. Due to the limited space, we have simplified the background and assumed that the reader has relevant background knowledge. We also note that other reviewers like Reviewer SakC and Reviewer 1FNr have commended the clarity of our paper's presentation. We will be more than willing to provide more details and revise our paper. Below are the answers to other raised questions.

Reviewer comment (Writing):
The related work seems to focus on listing many papers. It would help if the related work would attempt to explain some of the key concepts instead of just listing them.

Author response:
A2: Due to the limited paper length, it is impractical to exhaustively detail the key concepts of all referenced papers. We assume our readers have a basic understanding of the relevant background knowledge. Therefore, in the related work section, we mainly focus on discussing key points crucial for elucidating the concepts central to our paper.

Specifically, in "Deep Generative Models for Image Generation", Para. 1 aims to introduce the two-stage latent space paradigm proposed by VQ-VAE[1], while Para. 2 discusses the emergence of diffusion models and their integration with VQ-VAE[1].

In "Generative Pre-training for Image Representation Learning", we focus on the development of generative representation learning, particularly for recognition tasks.

The subsequent section, "Generative Modeling for Unifying Representation Learning and Image Generation", examines how previous studies have tackled the challenge of learning general representation for both generation and recognition tasks.

Additionally, at the end of each subsection, we articulate the relationships and distinctions between these approaches and our ADDP framework.

Reviewer comment (Presentation):
Figure text is often too small to read in print.

Author response:
A3: Thanks for the suggestion. The figure texts are enlarged in the revised version. 
In addition, we suggest relocating this comment to the 'Questions' section for appropriate categorization, because issues related to figure text formatting are not typically considered major weaknesses (as per the reviewing [instructions](https://iclr.cc/Conferences/2024/ReviewerGuide#Reviewing%20instructions) and [examples](https://iclr.cc/Conferences/2024/ReviewerGuide#Review%20examples) provided in the ICLR reviewer guidelines).

Reviewer comment (Novelty):
Are VQ-Tokens defined following van den Oord?

Author response:
A5: Yes. The VQ-Tokens in our paper corresponds to what is referred to as “latent variable” in VQ-VAE[1]. We adopt the term "VQ Tokenizer" to denote the VQ Encoder following Muse[3]. Accordingly, "VQ-Tokens" refers to the outputs produced by the VQ Tokenizer.

Reviewer comment (Writing):
What is the VQ-Decoder in equation 3? Which one is used?

Author response:
A6: The VQ-Decoder in Eq.(3) is identical to the one described in Eq.(1). We utilize the VQ-Decoder released by MAGE[2] in our implementation.

Reviewer comment (Reproducibility):
What is the encoder structure in equation two?

Author response:
A7: Our method imposes no constraints on the structure of the encoder. Theoretically, it can be any model capable of processing images as input and producing representations. In our experiments, we employed ViT-L, ViT-B and ResNet50 as encoders. For detailed results, please refer to Tab. 2, Tab. 7 and Tab. 8 respectively.

Reviewer comment (Theory):
Why can the MAGE-L network be discarded during inference?

Author response:
A9: As detailed in the text prior to Eq.(4), the token predictor (MAGE-L) can be discarded because it is only used to generate $q(\bar{z}_{t-1} | z_t)$, which our model has learnt to predict during training. Consequently, our model inherently incorporates the function of the token predictor during inference.

Reviewer comment (Presentation):
What do the lock symbols mean in Figure 4? Perhaps the locks signify constant network elements?

Author response:
A10: That’s correct. Utilizing the lock symbol to denote the frozen components of the network during training is a common practice. We apologize for not providing this clarification in the original figure caption and have rectified this oversight in the revised version.

Reviewer comment (Evaluation):
I am concerned that the current approach of utilizing VQ-AE for the diffusion process and the token-to-pixel conversion could diminish the generative diversity of the model.

Author response:
> **Q2: How to ensure the pixel-to-token generation process is not limited by a frozen VQ Encoder-Decoder? (Additional Evaluation) Quantitative comparison on the generation diversity.**

A2: Employing a frozen VQ Encoder-Decoder is a well-established practice for image generation, as evidenced by its adoption in previous works such as MAGE[2], MaskGIT[1], and Muse[3]. Therefore, the limitation (if exists) caused by the VQ Encoder-Decoder is widespread, and thus is not the main focus of this paper.

As for the evaluation of generation diversity, the FID used in our paper can capture both diversity and fidelity according to [5], where a lower FID indicates not only improved visual quality but also increased diversity. 

To further analyze diversity, we employ precision and recall metrics following ADM[5] as well. Higher recall indicates better diversity. The results are detailed in the table below.

Although the non-VQ model ADM[5] without classifier guidance achieves high diversity, its fidelity is significantly worse than ours. Besides that, our method achieves both better fidelity and diversity compared to ADM[5] with classifier guidance and MAGE[2] on unconditional generation. 

We have updated the experiment and discussion into appendix C.5 of our revised version.

| Method | FID $\downarrow$ | IS $\uparrow$ | Precision $\uparrow$ | Recall $\uparrow$ |
| --- | --- | --- | --- | --- |
| ADM (w/o classifier guidance) | 26.2 | 39.7 | 0.61 | 0.63 |
| ADM (w/ classifer guidance) | 12.0 | 95.4 | 0.76 | 0.44 |
| MAGE | 9.1 | 105.1 | 0.75 | 0.47 |
| Ours | 7.6 | 105.1 | 0.74 | 0.49 |

Reviewer comment (Experiments):
Could you provide more details on the criteria used to generate reliable and unreliable tokens? It is unclear which specific mechanisms within the model determine a token's reliability. An ablation study focusing on this aspect could offer deeper insights into the significance and impact of this feature.

Author response:
> **Q1: Details on the criteria used to generate reliable and unreliable tokens. (Additional Evaluation) Ablation study on the mechanism to determine reliability.**

A1: Thanks for the suggestions. As outlined in Appendix D.4, we generate reliable tokens using an iterative decoding strategy following MaskGIT[1] during inference. 

New reliable tokens are determined based on the predicted probability $p_\theta(\bar{z}_{t-1} | x_t)$ for a given timestep $t$. Specifically, for each masked location, we first sample a candidate token id and its corresponding probability $p$ from the predicted distribution. Then the confidence score $s$ of each location is calculated by adding a Gumbel noise $\epsilon$ to the log probability, i.e. $s = log(p) + \tau \epsilon$. Here, $\tau$ is set to be $6.0 \times \frac{t}{T}$ by default. Tokens with higher confidence scores are selected as the new reliable tokens, while those with lower scores remain to be masked for the next timestep.

During the training phase, without losing generalization, we opt to randomly sample the reliable tokens. The total number of reliable tokens is determined by the sampled timestep $t$. Specifically, the set of reliable tokens at step $t$ invariably includes the reliable tokens at step $t+1$.

We also conduct an additional ablation study to assess the impact of different mechanisms on determining token reliability during inference, as detailed in the tables presented below. Two key ablation choices are considered here: 

1. truncating the probability distribution employed for sampling each candidate token by using nucleus sampling[8], denoted by `top-p`;
2. varying the value of $\tau$ to adjust the randomness introduced by the Gumbel noise when computing the confidence scores.

| `top-p` | FID $\downarrow$ | IS $\uparrow$ |
| --- | --- | --- |
| 1.0 (default) | 7.6 | 105.1 |
| 0.95 | 7.9 | 117.3 |
| 0.9 | 10.5 | 124.1 |
| 0.7 | 34.1 | 88.6 |
| 0.5 | 90.9 | 32.9 |
| 0.3 | 185.8 | 8.4 |
| 0.1 | 325.8 | 2.3 |

| $\tau$ | FID $\downarrow$ | IS $\uparrow$ |
| --- | --- | --- |
| 0.0 | 353.7 | 1.3 |
| 2.0 | 18.5 | 107.2 |
| 6.0 | 12.1 | 80.7 |
| 20.0 | 27.3 | 48.7 |
| 60.0 | 33.8 | 41.0 |
| $2.0 \times \frac{t}{T}$ | 23.3 | 109.4 |
| $6.0 \times \frac{t}{T}$ (default) | 7.6 | 105.1 |
| $20.0 \times \frac{t}{T}$ | 17.7 | 63.9 |
| $60.0 \times \frac{t}{T}$ | 27.9 | 48.3 |

Our findings indicate that the current setup is nearly optimal, yielding the lowest FID. However, it's noteworthy that the IS score benefits from slightly reducing the value of `top-p` and $\tau`. This suggests that disregarding tokens with excessively low confidence may enhance the quality of synthesized images.

The discussions and experiments above have been added into the appendix C.4 of our revised version.

Reviewer comment (Experiments):
How can we ensure that the pixel-to-token generation process does not become constrained by a limited range of samples, especially when utilizing a frozen VQ Encoder-Decoder? Additionally, it would be beneficial to see quantitative comparisons of the proposed method against non-VQ generative models, specifically concerning the diversity of generated samples.

Author response:
> **Q2: How to ensure the pixel-to-token generation process is not limited by a frozen VQ Encoder-Decoder? (Additional Evaluation) Quantitative comparison on the generation diversity.**

A2: Employing a frozen VQ Encoder-Decoder is a well-established practice for image generation, as evidenced by its adoption in previous works such as MAGE[2], MaskGIT[1], and Muse[3]. Therefore, the limitation (if exists) caused by the VQ Encoder-Decoder is widespread, and thus is not the main focus of this paper.

As for the evaluation of generation diversity, the FID used in our paper can capture both diversity and fidelity according to [5], where a lower FID indicates not only improved visual quality but also increased diversity. 

To further analyze diversity, we employ precision and recall metrics following ADM[5] as well. Higher recall indicates better diversity. The results are detailed in the table below.

Although the non-VQ model ADM[5] without classifier guidance achieves high diversity, its fidelity is significantly worse than ours. Besides that, our method achieves both better fidelity and diversity compared to ADM[5] with classifier guidance and MAGE[2] on unconditional generation. 

We have updated the experiment and discussion into appendix C.5 of our revised version.

| Method | FID $\downarrow$ | IS $\uparrow$ | Precision $\uparrow$ | Recall $\uparrow$ |
| --- | --- | --- | --- | --- |
| ADM (w/o classifier guidance) | 26.2 | 39.7 | 0.61 | 0.63 |
| ADM (w/ classifer guidance) | 12.0 | 95.4 | 0.76 | 0.44 |
| MAGE | 9.1 | 105.1 | 0.75 | 0.47 |
| Ours | 7.6 | 105.1 | 0.74 | 0.49 |

Reviewer comment (Evaluation):
The performance gap observed between ADDP and other methodologies in the Linear ImageNet benchmark warrants a comprehensive explanation. Could you elucidate the detailed reasons behind this discrepancy?

Author response:
> **Q4: Detailed reason behind linear probing benchmark.**

A4: First, it is important to emphasize that the core capability of self-supervised learning (SSL) lies in fine-tuning, not linear-probing. Poor linear-probing should not negate the models. MAE [7] and BEiT [6] have highlighted this point. Still, we recognize that this is a potential direction for future research, and we will take this aspect into consideration in future work.

Nevertheless, we explored the reasons for our poor linear-probing. As discussed in Sec. 4.2, our model is trained on noisy synthetic images, making it hard in linear-probing on real images. We speculate that training on real images may help improve the linear-probing performance. To validate this, we fine-tuned our final model for 50 epochs with a 50% input mask ratio and $z_0$ as targets. The linear-probing result improved from 11.5% to 41.2% (better than BEiT), indicating that using data closer to real images indeed enhances linear-probing performance.

Reviewer comment (Evaluation):
Finally, visualizing the token spaces through projection to a 2D plane could offer a more intuitive understanding of the model's performance. Have you considered including such visualizations or projections to aid in the evaluation of the model's effectiveness?

Author response:
> **Q5: (Additional Evaluation) Visualizing the token space.**

A5: In Fig. 9 of the Appendix, we present the visualization of the noisy decoded images at each step, which can be viewed as the projection of token space into raw pixel space.

Additionally, in the revised version of our paper, we also update Fig. 16, which offers direct visualizations of the token space during generation as well. As is shown in Fig. 16, we map the top 3 PCA components of each token embedding (both reliable and unreliable) to RGB values. The visualization intuitively shows that, at each intermediate step, reliable tokens help sharpen the image and add more details, while unreliable tokens help refine the coarse-grained spatial contours. Collectively, these tokens maintain the spatial consistency of the generated image, highlighting the efficacy of our approach.