Paper: Emu: Generative Pretraining in Multimodality (ICLR-2024)

Reviewer comment (Experiments):
The comparison of image-text and video-text tasks should be also made with other state-of-the-art multimodal models, such as GPT-4V, PaLI-X (55B). It is okay to underperform given that those models are (likely) larger in the model size. The important part is to give reader a full picture of how this work compares to those important (private) models.

Author response:
**Q1: Comparision to PALI-X and GPT-4V**

We have added PaLI-X's results in our paper for comparison. We would like to clarify that the comparison may be unfair as PaLI-X is 1) much larger, 2) pretrained in-house data, 3) fine-tuned on each academic dataset rather than zero-shot. As for GPT-4V, we are unable to evaluate it because its API is just released several days ago and we have limited access.

Reviewer comment (Experiments):
I am curious if the pre-trained model can do more complex tasks such as the in-context subject-driven generation introduced in the paper [1]. Given that the model is general enough to handle prompts such as "Given the a few images about a common subject {subject_name}: {image_1}, {image_2}, {image_3}. Generate a new rendition of this same subject, in the Hall of Mirrors in Versailles.". It would be amazing if such complex task can be done in zero-shot.

[1] Subject-driven text-to-image generation via apprenticeship learning

Author response:
**Q2: Complex tasks like subject-driven image generation**

Thanks for the great suggestion. This is a challenging problem and the existing works use specialist architectures or specialist models for each personized subject. We show that our model can perform this task in some cases (as shown in the 4th row of Figure 1), but it is too hard for the current model to solve this problem proficiently. However, we think this is an interesting problem and we believe that after further scaling up the data and model sizes, this kind of zero-shot ability might emerge, given the flexibility of our pretraining framework.

Reviewer comment (Novelty):
The paper misses some key and very relevant comparative works like Cm3Leon (https://arxiv.org/abs/2309.02591), AnyMAL(https://arxiv.org/abs/2309.16058) etc. These papers should be compared against and explained how the authors work differes from the same.

Author response:
We thank the reviewer for providing the reference, and we included the performance comparison to these baselines below. We have updated a discussion and comparison in the paper (Section D). It is essential to note that the mentioned papers were released to arXiv less than one month before the ICLR submission deadline, e.g., Sept 5 for Cm3Leon, Sept 27 for AnyMAL, and Sept 28 for ICLR DDL, and our project was concluded prior to our awareness of these concurrent papers.

|                   | COCO FID  | COCO Caption | VQAv2    | OKVQA    | VizWiz   | VisDial  |
| ----------------- | --------- | ------------ | -------- | -------- | -------- | -------- |
| CM3Leon (Sep. 5) | **10.82** | 61.6         | 47.6     | 23.8     | 37.6 | 22.6     |
| AnyMAL-13B (Sep. 27) | - | 99.5         | 59.6     | 33.1     | 24.4 | -     |
| Emu               | 11.66     | 112.4    | 52.0 | 38.2 | 34.2     | 47.4 |
| Emu *               | -    | -    | 52.9 | 42.8 | 34.4  | 47.8 |
| Emu-I               | -     | **120.4**    | 57.2 | 43.4 | 32.2     | 43.0 |
| Emu-I *               |  -  | -    | **62.0** | **49.2** | **38.3**   | **51.1** |

Reviewer comment (Experiments):
It would be beneficial to see a discussion on the scalability of Emu with respect to the size and diversity of training data, and how the model might perform with fewer resources or less diverse data.

Author response:
We have added the experiments of smaller scale and less diverse data.

|                      | COCO   | OKVQA  | MSVDQA* |
| -------------------- | ------ | ------ | ------- |
|                      | 0-shot | 0-shot | 0-shot  |
| Emu-7B w/o YT-SB-1B  | 110.8  | 27.6   | 30.2    |
| Emu-7B w. YT-SB-1B   | 112.9  | 31.3   | 30.8    |
| Emu-14B w. YT-SB-1B  | 112.4  | 38.2   | 34.3    |
| Emu-14B w. 50k steps | **120.5**  | **41.6**   | **36.0**    |

 *the zero-shot prompt of MSVDQA is built by using two examples from the task

With scaling up the training data, i.e., the inclusion of additional data from YT-SB-1B, 7 billion parameters Emu demonstrates a noteworthy improvement in zero-shot performance on tasks such as OKVQA(+3.7), and MSVDQA(+0.6).  With an increase in model size to 14 billion parameters, zero-shot performance experience further enhancement. Notably, zero-shot performance gains +6.9 on OKVQA, and +3.5 on MSVDQA. This emphasizes the positive impact of scalability, considering both the size and diversity of training data, along with the increased model capacity, on the overall performance across a diverse set of tasks.

To further show the scalability of Emu, we scale up the  training schedule with more iterations and training samples, zero-shot performance gains +8.1 on COCO caption, +3.4 on OKVQA, and +1.7 on MSVDQA.

Reviewer comment (Experiments):
Autoregressive models are know to be notoriously slow at inference time due to the sequential nature of their execution. A comparative study of the same and ideas to make it better could help the paper.

Author response:
We agree that autoregressive model has its unique problems. However, we want to note that autoregressive model is the predominant architecture now and a lot of work to improve the inference speed have been devised, e.g. Flash Decoding. And these techniques also work in our case. Improving efficiency or speed is out of the scope of this paper, instead we offer a new perspective and approach to enable generative pretraining in multimodality. Please let us know if we misunderstood the question.

Reviewer comment (Writing):
How does causal transformer convert an image as multiple visual tokens. In section 2, is {z_1, z_2, ... z_N} the same as g(I).

Author response:
**Mechanism of Causal Transformer.** 

{z_1, z_2, ... z_N} is not the same as g(I). g(I) is the encoding of images, outputted from EVA-CLIP, which will be fed into the Causal Transformer to be transformed into causal tokens {z_1, z_2, ... z_N}. They are the input and output of Causal Transformer, respectively.

Each block of Causal Transformer consists of three layers: causal (masked) self-attention, cross-attention, and feed-forward layer. The process of aggregating information from 2D spatial visual signals into 1D causal sequence layer by layer is elaborated below:

- Causal self-attention layer: we feed N learnable tokens (vectors) into this layer as Q, K and V. The output tokens encode causal dependencies among each other.
- Cross-attention layer: the outputs from the previous causal self-attention layer serve as Q, while image features (g(I)) from EVA-CLIP serve as K and V. Each token can see all image features and extract information from them.
- The feed-forward layer: a standard feed-forward layer.

Through this process, the input vectors aggregate information from the image features while encoding causal dependency among each other, becomes {z_1, z_2, ... z_N}.

Reviewer comment (Reproducibility):
Is the N visual embeddings for the image decoder the same as the visual embedding after the causal transformer.

Author response:
**Is the N visual embeddings for the image decoder the same as those after the causal transformer?** No. The input of the image decoder is the embeddings generated by the Multimodal Modeling module.

Reviewer comment (Experiments):
EMU is trained with Laion-COCO, which has image caption in the style of COCO. How does that impact results in table 1? Is the zero-shot results as good if removing Laion-COCO?

Author response:
**Q2: Zero-shot concern about COCO evaluation**

We thank the reviewer for the comment. We first report the zero-shot results on NoCaps and Flickr30K captioning benchmarks, which are absolutely out-of-domain and render no concerns about zero-shot. The results show that Emu achieves superior performance over the baselines.

|             | NoCaps    | Flickr30K |
| ----------- | --------- | --------- |
| MetaLM      | 58.7      | 43.3      |
| KOSMOS-1    | -         | 67.1      |
| Flamingo-9B | -         | 61.5      |
| Emu         | 96.5      | 72.0      |
| Emu-I       | **108.8** | **77.4**  |

We report the results of removing LAION-COCO below. 
|                                           | COCO Cap  |
| ----------------------------------------- | --------- |
| w LAION COCO                              | 112.9     |
| w/o LAION COCO                            | 89.1      |
| w 20M GPT-reformulated data from LAION-2B | **117.0** |

The w/o LAION COCO experiment contains image-text pairs from only LAION-2B. LAION-2B is a large-scale dataset rich in world knowledge, but it contains poor captions, as exemplified by instances such as 'Sentry Gaming Headset, HPX-GX250B' and 'Angry Birds 2 Level 107 Shangham 3-Star Walkthrough'. Models pretrained exclusively on LAION-2B tend to generate text outputs of a similar nature, leading to a drop in CIDEr on the COCO caption benchmark, which specifically emphasizes the well-formed text. To encourage the learning of well-formed text, we included the synthetic dataset LAION-COCO during pretraining.

LAION-COCO serves as a valuable supplement to LAION-2B by providing examples of well-formed captions, thereby aiding models in generating more coherent and linguistically appropriate text. It's worth noting that LAION-COCO can be replaced by other datasets featuring well-formed text. For instance, by randomly sampling 20 million samples from LAION-2B and rewriting the captions using ChatGPT, the model exhibits a notable improvement in zero-shot performance on the COCO caption benchmark, even without LAION-COCO. This demonstrates the adaptability of the model to different well-formed text datasets for enhanced performance.

Reviewer comment (Experiments):
Other multimodal baselines for FID such as https://arxiv.org/abs/2309.02591 should be added to table 2.

Author response:
**Q3: Miss baselines.**

We thank the reviewer for pointing it out. We have incorporated a discussion and result comparison in our updated paper (Section D). It is noteworthy that our project was completed prior to our awareness of the contemporaneous work, CM3Leon [1], which was released to arXiv less than one month before the ICLR submission deadline.

In addition to the paper revision, we provide a comparison here. We can find that Emu outperform CM3Leon by a significant margin on most of tasks.

|                   | COCO FID  | COCO Caption | VQAv2    | OKVQA    | VizWiz   | VisDial  |
| ----------------- | --------- | ------------ | -------- | -------- | -------- | -------- |
| CM3Leon (Sept. 5) | **10.87** | 61.6         | 47.6     | 23.8     | **37.6** | 22.6     |
| Emu               | 11.66     | **112.4**    | **52.9** | **42.8** | 34.4     | **47.8** |

[1] Scaling Autoregressive Multi-Modal Models: Pretraining and Instruction Tuning

Reviewer comment (Theory):
As shown in table 2, there is performance loss due to regression to visual embeddings, instead of directly optimizing generating the best images.

Author response:
**Q4: Suboptimal image generation capability and performance loss in Table 2.**

We thank the reviewer for the question. We would like to clarify that this is probably because the condition space (image embeddings) of our visual decoder deviates a lot from the condition space (text embeddings) of the diffusion model used as initialization. Besides, our model is finetuned for a relatively short 15k steps, as discussed in Section 5.1.

Reviewer comment (Evaluation):
What is the FID before and after the visual decoding training?

Author response:
**Q5:  the FID before and after visual decoder training.**

The FID before visual decoding training is not meaningful because the condition space (image embeddings) of our visual decoder deviates a lot from the condition space (text embeddings) of the diffusion model used as initialization. Moreover, we can not calculate the FID directly without visual decoder training because of dimension mismatch between our image embeddings and text embeddings used in the original diffusion model. We report the intermediate FID in the training progress before the complete training here:

|           | COCO FID |
| --------- | -------- |
| 2k steps  | 17.05    |
| 5k steps  | 15.74    |
| 15k steps | 11.66    |