Paper: Bridging Autoregressive and Masked Modeling for Enhanced Visual Representation Learning (ICLR-2024)

Reviewer comment (Novelty):
The method is not compared with last year's SAIM paper [1]. SAIM and RandSAC are contemporaneous works that explore self-supervised approaches using autoregressive methods. The autoregressive self-supervised method has also shown good performance on downstream tasks. RandSAC has explored group-level reasoning. The paper should demonstrate the advantages of its method compared to SAIM's patch-based autoregressive and RandSAC's segment-based approaches, and address the question of whether the proposed method is merely a combination of these two methods.

Author response:
Thanks for bringing up this previous work! SAIM is also an excellent work building an autoregressive model within the CV field. However, SAIM only considers the patch-level autoregressive modeling and neglect the advantages brought from MIM. In contrast, we provide a unified perspective and effectively fuse the targets of both paradigms. **Our proposed GVP is not a mere combination of RandSAC and SAIM; rather, GVP represents a more generalized and effective modeling ways of images**. Here, we list three points on the comparison among our proposed method GVP, SAIM and RandSAC below:

1. **Unified perspective on the inter-patch dependency relationship**: Our formulation provides a unified perspective on the dependency relationships between MIM and autoregressive model. Based on this perspective, we formulate the fused targets, which are aimed to corporate the merits of MIM and autoregressive tasks. Therefore, we can benefit from both pretraining paradigms. On the other hand, **RandSAC and SAIM just treat their models as an autoregressive model and neglect the benefits that MIM may bring.** Indeed, our proposed GVP-p2p has better performance in classification tasks as shown in the table below:

|Model|Epochs|Linear Probing Acc.|Finetuning Acc.|
|---|---|---|---|
|RandSAC|1600|68.9|83.9|
|SAIM|800|62.5|83.9|
|GVP-p2p|800|69.6|84.0|

2. **Additional module $y_{extra}$**: Another improvement involves the introduction of the concept of $y_{extra}$ within our framework, enhancing the adaptability of our approach across diverse tasks. To substantiate this assertion, we conduct additional experiments to demonstrate that the incorporation of $y_{extra}$ can improve the performance on downstream tasks. The experimental settings are as follows: we feed the image into a pretrained VQGAN model to extract and pool the features of each patch. This pooled outcome represents a form of global information for the image, hereby designated as $y_{extra}$. The integration of $y_{extra}$ is applied to both the pretraining and downstream task stages in GVP-p2p variant, while other parameters remain constant. The results are shown in Appendix E. While the VQGAN feature alone does not exhibit strong linear probing performance, its integration into the process results in a notable improvement of 4%. This outcome underscores the positive impact of $y_{extra}$.

3. **Form of Input and Prediction Target**: In our paper, we explore diverse settings for the form of input and prediction Target, including pixel to pixel (p2p), pixel to discrete (p2d) and dicrete to discrete (d2d). **This exploration serves to highlight the flexibility and generality of our proposed methods.** In contrast, SAIM discusses about the Gaussian smoothing applied to the prediction target, which enhances the model's capacity to capture information about the low-frequency levels of the image. RandSAC only adopts the raw pixel of the patch to be the reconstruction target. **These two works overlook the diversity of input and prediction forms in the field of representation learning and, as a result, are not general enough.**

We have cited SAIM and added more discussions in Section 2 in the revision.

Reviewer comment (Evaluation):
The article does not describe the efficiency of the method. It is unclear whether the method significantly delays training time, thereby reducing training efficiency.

Author response:
We conduct experiments using 8 nodes, each of which contains 8 A100 GPU. The comparison of pretraining time on ImageNet with ViT-B for 200 epochs is listed below:

| Method | Pretraining time (hours) |
| -------- | -------- |
| MAE (p2p)   | 11.7 |
| BEiT (p2d)  | 13.2 |
| MAGE (d2d)  | 13.1 |
| GVP-p2p | 15.8|
| GVP-p2d | 16.7|
| GVP-d2d | 16.8|

In comparison to the baseline methods, our approach requires approximately 30% additional time within each input and target configuration. This increment in time is acceptable given the corresponding enhancement in performance.

Regarding inference time, the incorporation of the causal mask does not introduce any additional inference time overhead. Consequently, the inference time remains consistent with methods using ViT as the architecture, such as MAE and BEiT, as shown in the table below:

|Method|Inference Time using the ViT-B backbone over ImageNet-1K|
|---|---|
|MAE|63.1s|
|GVP-p2p|63.1s|

Regarding the batch size, the batch size of 4096 is consistent with several baselines such as MAE and RandSAC. We just follow this default batch size and do not mean to choose a large batch size.

Reviewer comment (Experiments):
The author uses a large batch size, which may be to obtain better performance. A more detailed analysis of this design choice should be provided.

Author response:
We conduct experiments using 8 nodes, each of which contains 8 A100 GPU. The comparison of pretraining time on ImageNet with ViT-B for 200 epochs is listed below:

| Method | Pretraining time (hours) |
| -------- | -------- |
| MAE (p2p)   | 11.7 |
| BEiT (p2d)  | 13.2 |
| MAGE (d2d)  | 13.1 |
| GVP-p2p | 15.8|
| GVP-p2d | 16.7|
| GVP-d2d | 16.8|

In comparison to the baseline methods, our approach requires approximately 30% additional time within each input and target configuration. This increment in time is acceptable given the corresponding enhancement in performance.

Regarding inference time, the incorporation of the causal mask does not introduce any additional inference time overhead. Consequently, the inference time remains consistent with methods using ViT as the architecture, such as MAE and BEiT, as shown in the table below:

|Method|Inference Time using the ViT-B backbone over ImageNet-1K|
|---|---|
|MAE|63.1s|
|GVP-p2p|63.1s|

Regarding the batch size, the batch size of 4096 is consistent with several baselines such as MAE and RandSAC. We just follow this default batch size and do not mean to choose a large batch size.

Reviewer comment (Experiments):
The current paper does not provide results for the multi-object visual scene detection task on COCO. It is recommended to include experimental results for downstream validation tasks related to object detection.

Author response:
Following your suggestion, we conduct object detection experiments on the MS COCO dataset [1]. We adopt ViT as the backbone of Mask-RCNN [2], following the architecture of ViT Benchmarking [3]. Due to the limited time, we only finetune the model for 25 epochs. We show the performance of representations learned through different self-supervised methods and supervised training. We report box AP for object detection, where the result of ViT-iGPT is borrowed from [4].

|Method|Pretraining Epochs|Finetuning Epochs|COCO-AP$^{bb}$|
|---|---|---|---|
|ViT-iGPT|300|100|45.5|
|MAE|200|25|46.2|
|GVP-p2p (ours)|200|25|46.6|

We observe that our method achieves 46.6 bbox mAP with 25 epochs of finetuning, which is better than autoregressive method iGPT and MIM method MAE. The results indicate that our method can benefit from both MIM and autoregressive methods and learn high-quality representations. We have added this experiment in Appendix G in the revision.

[1] Lin et al. Microsoft COCO: Common Objects in Context, Arxiv 1405.0312.

[2] He et al. Mask R-CNN, Arxiv 1703.06870.

[3] Li et al. Benchmarking Detection Transfer Learning with Vision Transformers, Arxiv 2111.11429

[4] Qi et al. Exploring Stochastic Autoregressive Image Modeling for Visual Representation, AAAI 2023.

Reviewer comment (Evaluation):
The paper should provide the inference time and training time.

Author response:
We conduct experiments using 8 nodes, each of which contains 8 A100 GPU. The comparison of pretraining time on ImageNet with ViT-B for 200 epochs is listed below:

| Method | Pretraining time (hours) |
| -------- | -------- |
| MAE (p2p)   | 11.7 |
| BEiT (p2d)  | 13.2 |
| MAGE (d2d)  | 13.1 |
| GVP-p2p | 15.8|
| GVP-p2d | 16.7|
| GVP-d2d | 16.8|

In comparison to the baseline methods, our approach requires approximately 30% additional time within each input and target configuration. This increment in time is acceptable given the corresponding enhancement in performance.

Regarding inference time, the incorporation of the causal mask does not introduce any additional inference time overhead. Consequently, the inference time remains consistent with methods using ViT as the architecture, such as MAE and BEiT, as shown in the table below:

|Method|Inference Time using the ViT-B backbone over ImageNet-1K|
|---|---|
|MAE|63.1s|
|GVP-p2p|63.1s|

Regarding the batch size, the batch size of 4096 is consistent with several baselines such as MAE and RandSAC. We just follow this default batch size and do not mean to choose a large batch size.

Reviewer comment (Evaluation):
Clarify whether the GPU usage is higher for this method compared to the RandSAC and SAIM methods.

Author response:
As our proposed method employs an architecture requiring two streams of forward propagation, aligning with the design of RandSAC and SAIM methods, the GPU usage remains consistent with that of RandSAC and SAIM.

Reviewer comment (Experiments):
I understand computation could be a problem, but would like to see more solid comparisons. E.g., the main table (Table 1) used a pre-training epoch of 200. At this stage, it is unclear whether the pre-training is just worse in upper bound, or it is just some pre-training converges faster. Plus, it would be much more fair to me if the comparison is done based on the "overall compute" or "overall data" spent on pre-training, rather than simply put number of epochs there. For example, the default number of epochs used in MAE is 800-ep, this is when the model has sufficiently converged. And even then, from 800 to 1600 there is still notable improvements in the final table. Comparing with MAE at 200-epoch is not the fair-most comparison.

Author response:
A1. We have included experiments with 800 epochs pretraining on ImageNet in Appendix C. We will also show the results in the following table:
|Method|Epochs|LP Acc.|FT Acc.|
|---|---|---|---|
|BEiT|800|56.7|83.2|
|SimMIM|800|56.7|83.8|
|MAE|800|65.1|83.6|
|RandSAC|1600|68.9|83.9|
|GVP-p2p|800|**69.6**|**84.0**|
|GVP-p2d|800|67.4|83.8|

We outperform the three MIM methods BEiT, SimMIM and MAE both in linear probing and finetuning results. It is noteworthy that our methods are on par with the autoregressive model RandSAC but we have much fewer pretraining epochs (Ours 800 epochs v.s. RandSAC 1600 epochs).

Reviewer comment (Theory):
(minor) I think Eq 3 has a typo that z should be greater or equal to m, instead of t.

Author response:
A2. Thanks for pointing it out. We have corrected it in the revision.

Reviewer comment (Reproducibility):
Is there a planned code release? I guess from XL-net it is fine but would be great to promote open-sourcing here.

Author response:
A3. Sure, we will definitly release the code upon acceptance.

Reviewer comment (Theory):
The motivation is not clearly illustrated. The paper claimed that “autoregressive models and masked modeling can complement each other” in Paragraph 3 of the introduction. However, I’m not aware of why “naturally generate sequences based on a specific masking pattern” is a good property. In fact, as the authors have stated, predicting autoregressively is not a natural choice in computer vision. Then, what’s the benefit of combining autoregressive models?

Author response:
We now understand your point on clarifying the benefit of autoregressive model. Indeed, as you mentioned, conventional autoregressive methods such as iGPT, face a challenge in that the fixed raster order they adopt is not a natural choice. This limitation makes the model incapable of effectively capturing bidirectional information. However, **autoregressive models enjoy the advantage of effectively capturing intricate dependency relationships between tokens.** This stands in contrast to the approach in MIM, where the predicted tokens are treated independently. As discussed in Section 3, this independence in token prediction can result in lower generation quality. 

Given the pros and cons of the two paradigms, **our motivation is to create a synergy between them by developing a novel modeling method.** We hope that the introduction of modeling intricate dependency by autoregressive modeling will assist MIM in learning more inter-image information and enhance its performance in generation tasks. In fact, by incoporating autoregressive modeling, GVP demonstrates improved image reconstruction ability compared to MAE, as evidenced in Appendix F. **We clarify the advantage of autoregressive model and make the motivation clearer in the introduction part in the revision.**

Reviewer comment (Novelty):
The design of two-stream attention is not well motivated. Previous methods like MAE use one-stream attention. The proposed framework does not put a restriction on the two-stream attention mechanism. What is the purpose of introducing two-stream attention?

Author response:
We understand your concern on the motivation and demonstration on the two-stream attention. We will clarify the motivation and present some details of two-stream attention both here and in the revision.

**The purpose of introducing two-stream attention.** As has been discussed in the backbone network part in Section 4.1, we are not capable of modeling with arbitrary index sequence in an autoregressive way if we use a single stream in ViT with causal masks. This is because of two requirements that are contradictory in a standard ViT architecture, which have been discussed in XLNet [1]: (1) to predict the token $x_{z_t}$, the network output in the $z_t$ position in one layer $g_{\theta}(x_{\mathbf{z}<t}, z_t)$ should only use the position $z_t$ and not the content $x_{z_t}$, (2) to predict the other tokens $x_{z_j}$ with $j > t$, $g_{\theta}(x_{\mathbf{z}<t}, z_t)$ should also encode the content $x_{z_t}$ to provide full contextual information. Therefore, it is not enough with only one stream. Since the two-stream attention proposed by XLNet facing the same problem works well, we adopt the two-stream attention proposed to solve this contradiction.

**Introduction of the two-stream attention.** The differences between content stream and query stream are shown below:
* The content representation $h_{\theta}(x_{\mathbf{z}\leq t})$ serves a similar role to the standard hidden states in ViT. This representation encodes both the condition $x_{\mathbf{z}< t}$ and $x_{z_t}$ itself. Therefore, the attention masks for the content stream should ensure that $x_{z_t}$ can be attended to itself. The content stream satisfies the second requirement in the above discussion.
* The query representation $g_{\theta}(x_{z<t}; z_t)$ only has access to the condition $x_{\mathbf{z}< t}$ and the position $z_t$, but not the content $x_{z_t}$. Therefore, the attention masks for the query stream should ensure that $x_{z_t}$ cannot be attended to itself. The query content stream satisfies the first requirement in the above discussion.

Therefore, the query, key and value in the content stream attention block are all from itself to fully encode contextual information. On the other hand, the key and value in the query stream attention block are from the content stream but the query is from the query stream itself to provide positional information. At last, the output from the query stream is seemed as the prediction result.

We have added these explanations in Section 4.1 and 4.2 in the revision to achieve better presentation.

[1] Yang et al. XLNet: Generalized Autoregressive Pretraining for Language Understanding, Arxiv 1906.08237.

Reviewer comment (Writing):
Two-stream attention is not well presented. What’s the differences between content stream and query stream? Why the causal masks are different for these two streams?

Author response:
We understand your concern on the motivation and demonstration on the two-stream attention. We will clarify the motivation and present some details of two-stream attention both here and in the revision.

**The purpose of introducing two-stream attention.** As has been discussed in the backbone network part in Section 4.1, we are not capable of modeling with arbitrary index sequence in an autoregressive way if we use a single stream in ViT with causal masks. This is because of two requirements that are contradictory in a standard ViT architecture, which have been discussed in XLNet [1]: (1) to predict the token $x_{z_t}$, the network output in the $z_t$ position in one layer $g_{\theta}(x_{\mathbf{z}<t}, z_t)$ should only use the position $z_t$ and not the content $x_{z_t}$, (2) to predict the other tokens $x_{z_j}$ with $j > t$, $g_{\theta}(x_{\mathbf{z}<t}, z_t)$ should also encode the content $x_{z_t}$ to provide full contextual information. Therefore, it is not enough with only one stream. Since the two-stream attention proposed by XLNet facing the same problem works well, we adopt the two-stream attention proposed to solve this contradiction.

**Introduction of the two-stream attention.** The differences between content stream and query stream are shown below:
* The content representation $h_{\theta}(x_{\mathbf{z}\leq t})$ serves a similar role to the standard hidden states in ViT. This representation encodes both the condition $x_{\mathbf{z}< t}$ and $x_{z_t}$ itself. Therefore, the attention masks for the content stream should ensure that $x_{z_t}$ can be attended to itself. The content stream satisfies the second requirement in the above discussion.
* The query representation $g_{\theta}(x_{z<t}; z_t)$ only has access to the condition $x_{\mathbf{z}< t}$ and the position $z_t$, but not the content $x_{z_t}$. Therefore, the attention masks for the query stream should ensure that $x_{z_t}$ cannot be attended to itself. The query content stream satisfies the first requirement in the above discussion.

Therefore, the query, key and value in the content stream attention block are all from itself to fully encode contextual information. On the other hand, the key and value in the query stream attention block are from the content stream but the query is from the query stream itself to provide positional information. At last, the output from the query stream is seemed as the prediction result.

We have added these explanations in Section 4.1 and 4.2 in the revision to achieve better presentation.

[1] Yang et al. XLNet: Generalized Autoregressive Pretraining for Language Understanding, Arxiv 1906.08237.

Reviewer comment (Experiments):
The ablation studies do not explain why the general framework performs better. Ablation studies should be responsible for telling why the performance is better. In order to do this, the authors need rigorous comparison experiments, especially with masked modeling paradigm. For example, experiments can begin with a pure masked modeling version (no grouping, no causal masks, no two-stream attention), and gradually add these modifications step by step to show what is key factor that contributes to the performance gain.

Author response:
Thanks for the suggestion! Following your suggestion, we conduct ablation study by incrementally adding the modifications. The models are pretrained using ViT-B with 200 epochs on ImageNet-1k. The results are presented in the table below:
|Modification|Linear Accuracy|Finetuning Accuracy|
|---|---|---|
|Pure Masked Modeling|52.5|82.8|
|+Causal Masks|28.4|78.4|
|+Causal Masks+Two-stream Attention|52.2|82.3|
|+Causal Masks+Two-stream Attention+Mixed Length Grouping|61.4|83.2|

From the results, we observe that (1) Directly adding causal masks on masked modeling introduces the problem that the model is unaware of the next position it will predict, as discussed in A2. This problem will lead to deteriorated performance. Leveraging the two-stream attention mechanism mitigates this issue. (2) Without a well-designed grouping method, the performance still remains consistent with the baseline. However, with the introduction of mixed length grouping, the performance surpasses the baseline by a large margin. This underscores the significance of a well designed grouping method in the modeling. We have added this ablation study in Appendix H.

Reviewer comment (Writing):
Notation is chaotic. Equation (2), $x$ is of different styles and inconsistent subscripts. Notation $m$ is not introduced.

Author response:
Thanks for point them out! We apologize for any confusion caused by these typos. They have been fixed in the revised version.

Reviewer comment (Writing):
Notation is chaotic. Page 6, Section 4.2, Para 2, Line 4, $M$ is not introduced.

Author response:
Thanks for point them out! We apologize for any confusion caused by these typos. They have been fixed in the revised version.

Reviewer comment (Experiments):
What is the group segmentation distribution during training?

Author response:
The group segmentation setting is chosen to be the mixed length setting with $K=20$ during training in the experiments shown in Table 1,2 and 3. Table 4 exhibits the performance under different group segmentation distributions.

Reviewer comment (Experiments):
Do you have the results on COCO detection task?

Author response:
We conduct object detection experiments on the MS COCO dataset [2]. We adopt ViT as the backbone of Mask-RCNN [3], following the architecture of ViT Benchmarking [4]. Due to the limited time, we only finetune the model for 25 epochs. We show the performance of representations learned through different self-supervised methods and supervised training. We report box AP for object detection, where the result of ViT-iGPT is borrowed from [5].

|Method|Pretraining Epochs|Finetuning Epochs|COCO-AP$^{bb}$|
|---|---|---|---|
|ViT-iGPT|300|100|45.5|
|MAE|200|25|46.2|
|GVP-p2p (ours)|200|25|46.6|

We observe that our method achieves 46.6 bbox mAP with 25 epochs of finetuning, which is better than autoregressive method iGPT and MIM method MAE. The results indicate that our method can benefit from both MIM and autoregressive methods and learn high-quality representations. We have added this experiment in Appendix G in the revision.


[2] Lin et al. Microsoft COCO: Common Objects in Context, Arxiv 1405.0312.

[3] He et al. Mask R-CNN, Arxiv 1703.06870.

[4] Li et al. Benchmarking Detection Transfer Learning with Vision Transformers, Arxiv 2111.11429

[5] Qi et al. Exploring Stochastic Autoregressive Image Modeling for Visual Representation, AAAI 2023.

Reviewer comment (Evaluation):
The presented method appears to cost at least 2x to train compared to alternative methods (due to processing tokens once for content and once for query). By failing to present and discuss compute cost, the comparison with existing works in the main text does not provide sufficient evidence to claim outperforming them.

Author response:
**Pretraining Time.** We conduct experiments using 8 nodes, each of which contains 8 A100 GPU. The comparison of pretraining time on ImageNet with ViT-B for 200 epochs is listed below:

| Method | Pretraining time (hours) |
| -------- | -------- |
| MAE (p2p)   | 11.7 |
| BEiT (p2d)  | 13.2 |
| MAGE (d2d)  | 13.1 |
| GVP-p2p | 15.8|
| GVP-p2d | 16.7|
| GVP-d2d | 16.8|

In comparison to the baseline methods, our approach requires approximately 30% additional time within each input and target configuration. This increment in time is acceptable given the corresponding enhancement in performance.

**Inference Time.** Regarding inference time, the incorporation of the causal mask does not introduce any additional inference time overhead. Consequently, the inference time remains consistent with methods using ViT as the architecture, such as MAE and BEiT.

**Memory.** As for the memory, the memory cost is consistent with existing autoregressive methods like RandSAC [1] and SAIM [2], which also need two streams in the forward propagation.

[1] Hua et al. Self-supervision through Random Segments with Autoregressive Coding (RandSAC), CVPR 2023.

[2] Qi et al. Exploring Stochastic Autoregressive Image Modeling for Visual Representation, AAAI 2023.

Reviewer comment (Evaluation):
Besides compute cost, the memory requirements also change and may make this method less interesting than existing approaches.

Author response:
**Pretraining Time.** We conduct experiments using 8 nodes, each of which contains 8 A100 GPU. The comparison of pretraining time on ImageNet with ViT-B for 200 epochs is listed below:

| Method | Pretraining time (hours) |
| -------- | -------- |
| MAE (p2p)   | 11.7 |
| BEiT (p2d)  | 13.2 |
| MAGE (d2d)  | 13.1 |
| GVP-p2p | 15.8|
| GVP-p2d | 16.7|
| GVP-d2d | 16.8|

In comparison to the baseline methods, our approach requires approximately 30% additional time within each input and target configuration. This increment in time is acceptable given the corresponding enhancement in performance.

**Inference Time.** Regarding inference time, the incorporation of the causal mask does not introduce any additional inference time overhead. Consequently, the inference time remains consistent with methods using ViT as the architecture, such as MAE and BEiT.

**Memory.** As for the memory, the memory cost is consistent with existing autoregressive methods like RandSAC [1] and SAIM [2], which also need two streams in the forward propagation.

[1] Hua et al. Self-supervision through Random Segments with Autoregressive Coding (RandSAC), CVPR 2023.

[2] Qi et al. Exploring Stochastic Autoregressive Image Modeling for Visual Representation, AAAI 2023.

Reviewer comment (Novelty):
This work provides little discussion, results, or ideas on how the added sampling flexibility can be used downstream, and it reads more like: "here is another (more complex) way to obtain very similar image representations".

Author response:
Indeed, **the added flexibility on the network sampling is usable for downstream tasks.** Our proposed flexible network design integrates the modeling paradigm from both mask modeling and autoregressive tasks, which is proved to be beneficial to various downstream tasks in the field of NLP [6]. As discussed in [6], the flexible attention masks enable the pretrained model suitable for both classification task and generation task. We believe that the flexible modeling patterns among patches can also bring benefits to various downstream tasks, which can be demonstrated by the following experimental results:

1. In the classification task, our proposed GVP achieves better performance than both MIM and autoregressive baselines that only take one modeling paradigm. Additionally, as one part of the model, the flexible attention mask ensures the alignment between the input distribution in the pretraining stage and the linear probing stage, which is helpful to the generalization (linear accuracy on ImageNet: with attention mask 61.5 v.s. without attention mask 56.2).

2. In the image reconstruction task, both conventional autoregressive methods and MIM methods exhibit shortcomings. When provided with patches at random positions within an image, conventional autoregressive methods like iGPT struggle to capture bidirectional information, limiting their ability to leverage complete information from the given patches. On the other hand, MIM methods like MAE may overlook the dependency relations between predicted patches, potentially encountering multimodality issues as discussed in [7], resulting in lower-quality generation. In contrast, our proposed methods GVP can leverage the added flexibility to enhance the generation performance. With the given patches as the condition information, we can perform the proposed group autoregressive modeling, generating predicted patches group by group. This approach will enjoy better reconstruction quality than MAE. Details can be viewed in Appendix F.

Following your suggestions, we have added the above discussion on how the flexibility is usable for downstream tasks in Section 5 and Appendix F. We hope that this discussion will demonstrate the superiority of our proposed framework on benefitting downstream tasks.

[6] Dong et al. Unified Language Model Pre-training for Natural Language Understanding and Generation, NeurIPS 2019.

[7] Gu et al. Non-autoregressive neural machine translation. ICLR 2018.

Reviewer comment (Evaluation):
What is the training cost of the presented method vs existing methods?

Author response:
**Pretraining Time.** We conduct experiments using 8 nodes, each of which contains 8 A100 GPU. The comparison of pretraining time on ImageNet with ViT-B for 200 epochs is listed below:

| Method | Pretraining time (hours) |
| -------- | -------- |
| MAE (p2p)   | 11.7 |
| BEiT (p2d)  | 13.2 |
| MAGE (d2d)  | 13.1 |
| GVP-p2p | 15.8|
| GVP-p2d | 16.7|
| GVP-d2d | 16.8|

In comparison to the baseline methods, our approach requires approximately 30% additional time within each input and target configuration. This increment in time is acceptable given the corresponding enhancement in performance.

**Inference Time.** Regarding inference time, the incorporation of the causal mask does not introduce any additional inference time overhead. Consequently, the inference time remains consistent with methods using ViT as the architecture, such as MAE and BEiT.

**Memory.** As for the memory, the memory cost is consistent with existing autoregressive methods like RandSAC [1] and SAIM [2], which also need two streams in the forward propagation.

[1] Hua et al. Self-supervision through Random Segments with Autoregressive Coding (RandSAC), CVPR 2023.

[2] Qi et al. Exploring Stochastic Autoregressive Image Modeling for Visual Representation, AAAI 2023.

Reviewer comment (Reproducibility):
What is the source for the numbers in Table 1? How were hyperparameters tuned for those?

Author response:
The results of iGPT and MaskGIT are from their original papers. For the remaining methods, we implement them by ourselves using their official code. The learning rate is searched within the set $\{0.5x, x, 2x\}$, where $x$ represents the default learning rate provided by the respective methods. Our obtained results align consistently with those reported in prior studies [3][4][5].

[3] Li et al. MAGE: MAsked Generative Encoder to Unify Representation Learning and Image Synthesis, CVPR 2023.

[4] Zhang et al. How Mask Matters: Towards Theoretical Understandings of Masked Autoencoders, NIPS 2022.

[5] Xie et al. Masked Frequency Modeling for Self-Supervised Visual Pre-Training. ICLR 2023.

Reviewer comment (Evaluation):
Is the added flexibility, in how these networks can be sampled, usable for downstream tasks?

Author response:
Indeed, **the added flexibility on the network sampling is usable for downstream tasks.** Our proposed flexible network design integrates the modeling paradigm from both mask modeling and autoregressive tasks, which is proved to be beneficial to various downstream tasks in the field of NLP [6]. As discussed in [6], the flexible attention masks enable the pretrained model suitable for both classification task and generation task. We believe that the flexible modeling patterns among patches can also bring benefits to various downstream tasks, which can be demonstrated by the following experimental results:

1. In the classification task, our proposed GVP achieves better performance than both MIM and autoregressive baselines that only take one modeling paradigm. Additionally, as one part of the model, the flexible attention mask ensures the alignment between the input distribution in the pretraining stage and the linear probing stage, which is helpful to the generalization (linear accuracy on ImageNet: with attention mask 61.5 v.s. without attention mask 56.2).

2. In the image reconstruction task, both conventional autoregressive methods and MIM methods exhibit shortcomings. When provided with patches at random positions within an image, conventional autoregressive methods like iGPT struggle to capture bidirectional information, limiting their ability to leverage complete information from the given patches. On the other hand, MIM methods like MAE may overlook the dependency relations between predicted patches, potentially encountering multimodality issues as discussed in [7], resulting in lower-quality generation. In contrast, our proposed methods GVP can leverage the added flexibility to enhance the generation performance. With the given patches as the condition information, we can perform the proposed group autoregressive modeling, generating predicted patches group by group. This approach will enjoy better reconstruction quality than MAE. Details can be viewed in Appendix F.

Following your suggestions, we have added the above discussion on how the flexibility is usable for downstream tasks in Section 5 and Appendix F. We hope that this discussion will demonstrate the superiority of our proposed framework on benefitting downstream tasks.

[6] Dong et al. Unified Language Model Pre-training for Natural Language Understanding and Generation, NeurIPS 2019.

[7] Gu et al. Non-autoregressive neural machine translation. ICLR 2018.