Paper: DreamLLM: Synergistic Multimodal Comprehension and Creation (ICLR-2024)

Reviewer comment (Evaluation):
For free-form interleaved generation, it is important to ensure the consistency between related images. However, as we can view the model as replacing the CLIP text encoder of stable diffusion with a much more powerful LLM for the image generation aspect, the control of the generated image is still limited. As we can see from the Figure 3, the phones in generated samples have large discrepancy.

Author response:
**Clarification.** We highly appreciate your insights into this perceived limitation. In fact, the proposed DreamLLM model architecture is able to generate highly consistent images. However, the images in the interleaved MMC4 dataset used to train DreamLLM inherently have low consistency as it is collected from the noisy Internet. This factor could potentially compromise the model's ability to fully manifest an emergent image consistency property.

**Evidence from subject-driven generation.** To substantiate our assertion that data with superior image consistency can fully leverage the capabilities of DreamLLM in generating consistent images, we have conducted additional experiments focusing on subject-driven image generation. This type of image generation necessitates the model to comprehend intricate instructions while concurrently preserving the subject identity features with utmost precision. Our DreamLLM has been fine-tuned for image-to-image generation in the third stage of SFT using the same data employed by BLIP-Diffusion [1], a recently proposed model in the domain of subject-driven generation. As this data set is not publicly available, we have constructed it independently based on the procedures outlined in the original paper. The results are shown in **Appendix A.4, Fig.8, page 25**. The results compellingly validate the effectiveness and great potential of applying DreamLLM for image-consistent generation.

[1] Li et al. BLIP-Diffusion: Pre-trained Subject Representation for Controllable Text-to-Image Generation and Editing. In NeurIPS 2023.

Reviewer comment (Experiments):
Ablation studies on the choices, combination ways, and the importance of filtering process of the training datasets are not shown, which might provide insights for future study.

Author response:
Thanks for your suggestions! Due to time limitations, it is impractical to perform a comprehensive ablation study encompassing all training datasets and filtering strategies, as such an undertaking may indeed exceed the scope of the current research. However, during rebuttal, we have conducted some ablation studies on instruction-following data and training data. We kindly refer you to our response to **Q1.a of Reviewer h8i6** for details. In addition, please note that the filtering strategies on datasets such as MMC4 are basically intuitive. During our manual review of the data, we identified several instances of misalignment between images and their corresponding text, which served as an impetus for our filtering approach. It is our future work to identify and resolve all data-related discrepancies.

Reviewer comment (Experiments):
Will including samples from training datasets as in-context examples improve the performance during evaluation?

Author response:
Indeed. We extend our gratitude for your insightful suggestion! Please refer to our [General Response](https://openreview.net/forum?id=y01KGvd9Bw&noteId=xe4TvtyFfS) for the in-context learning and few-shot experimental results, the results demonstrate the powerful in-context learning and few-shot learning capabilities of DreamLLM.

Reviewer comment (Experiments):
Several datasets used by the model are derived from COCO datasets, e.g., Laion-COCO, LLaVaInstruct, etc. It is unclear how to determine if there is data leakage in the training datasets. This concern applies to the results in Table 1 as well as Table 2.

Author response:
**Creation. (Tab. 2)** There is **no** data leakage for the evaluation. **No images from COCO are used as creation targets during training.** We want to emphasize that `LAION-COCO` or `BLIP-LAION` images are from the LAION dataset, but it is constructed by replacing the original noisy Internet captions with captions made by COCO-pretrained model BLIP. To verify if the COCO-style captions benefit downstream COCO FID, we conduct experiments by fine-tuning SDv2.1 on the same training dataset as DreamLLM. The results are shown in the following table. Note that SDv2.1 is pretrained on LAION-5B with unknown details, and all models below are based on a pretrained SDv2.1. From the results, it can be observed that the training data used by DreamLLM indeed benefits SDv2.1 on COCO text-to-image FID. This may come from the COCO-style captions, which are close to the COCO dataset. However, our DreamLLM still outperforms the SDv2.1 baseline by a clear margin.

| Model              | Data                                                       | MS-COCO  | LN-COCO   |
| ------------------ | ---------------------------------------------------------- | -------- | --------- |
| SDv2.1             | Unknown Details                                            | 12.43    | 34.26     |
| SDv2.1             | `LAION400M` (11M) + `BLIP-LAION` (8M) + `LAION-COCO` (11M) | 11.91    | 25.35     |
| DreamLLM-7B (Ours) | `LAION400M` (11M) + `BLIP-LAION` (8M) + `LAION-COCO` (11M) | **8.46** | **20.53** |

Reviewer comment (Reproducibility):
The implementation of the multi-token dream query is unclear. For decoder-only transformer training, every token needs a loss (or a score). What is the loss of each query token during training, and are they generated sequentially or altogether during inference?

Author response:
> **Q2.a:** How is the multi token dream query implemented? For decoder-only transformer training, every token needs a loss (or a score). What is the loss of each query token during training, and are they generated sequentially or altogether during inference?

- The multi-token dream query is implemented as learnable parameters, i.e., `nn.Parameter()`. 
- During training, only the text tokens (including some special tokens such as the \<dream\> tokens) need a per-token language modeling loss (i.e., Cross-Entropy loss). The query tokens are fed **altogether** into the SD image decoder, and it is supervised by directly optimizing the **diffusion loss** (i.e., noise prediction MSE loss).
- During inference, the pretrained dream queries are fed altogether into the LLM when the special \<dream\> token is generated. The dream queries do not need to be generated since they are learnable parameters that are part of DreamLLM.

Reviewer comment (Reproducibility):
For stage 1, stage 2, and stage 3, what is the final loss? Do they both have L_DM (formula 5) and L_MLLM (formula 6)? Is there a weight?

Author response:
> **Q2.c:** For the stage1, stage2, stage3, what's the final loss? Do they both have L\_DM (formula 5) and L\_MLLM (formula 6)? Is there a weight?

Sorry for the confusion about the training details. The detailed training implementation is listed in **the first table** in our response to **Q1.a**. $\mathcal{L}\_{\text{DM}}$ is used for training `creation` , while $\mathcal{L}\_{\text{MLLM}}$ is used for training `comprehension`. If the objective is `both`, then we sum the two losses with a weight $\alpha$: $\mathcal{L}\_{\text{MLLM}} + \alpha\mathcal{L}\_{\text{DM}}$. We set $\alpha=10$ in this work to balance the loss.

Reviewer comment (Reproducibility):
For the I-GPT training, does the visual projector, condition projector, and dream embedding get updated, or is only the MLLM transformer updated?

Author response:
> **Q2.d:** For the I-GPT training, does the visual projector, condition projector, dream embedding get updated? or only the MLLM transformer got updated?

Sorry for the confusion about the training details. During $\mathcal{I}$-GPT training, all model components are trained jointly, including the visual projector, condition projector, dream embeddings, and the LLM. We have presented a more detailed training configuration in **the first table** in our response to **Q1.a**.

Reviewer comment (Experiments):
Can the model do k-shot learning for image understanding tasks?

Author response:
> **Q3:** Can the model do k-shot learning for image understanding task?

Thank you for your constructive suggestion!  Please refer to our [General Response](https://openreview.net/forum?id=y01KGvd9Bw&noteId=xe4TvtyFfS) for the in-context learning and few-shot experimental results, the results demonstrate the powerful in-context learning and few-shot learning capabilities of DreamLLM.