This approach is reflected in subsequent high-quality image generation frameworks, such as DALLE 3 and SORA, which leverage the generation and understanding capabilities of VLMs to obtain higher-quality text conditions for generation tasks. Text-image Instructions. For better cross-modality alignment and instruction finetuning, we collect high-quality datasets from publicly available sources. In particular, for cross-modality alignment, we utilize 558K image-caption pairs from the LLaVA-filtered CC3M dataset and 695K sampled GPT-4V-responded captions from the ALLaVA dataset. It brings about 1.2M image captions in total for projector pretraining. As for instruction finetuning, we sample 643K single- and multi-turn conversations (excluding 21K TextCaps data) from the LLaVA dataset, 100K QA pairs from ShareGPT4V, 10K LAION-GPT-4V captions, 700K GPT-4V-responded instruction pairs from ALLaVA dataset, and 6K text-only multi-turn conversations from LIMA and OpenAssistant2. To bolster the OCR-related abilities, we further collect 28K QA pairs that comprise 10K DocVQA, 4K ChartQA, 10K DVQA, and 4K AI2D data. In general, there are about 1.5M instruction-related conversations for image comprehension. Moreover, we also collect 13K pairs for image-related generation that will be elaborated on subsequently. Generation-related Instructions. To support image generation, we further construct a 13K instruction-following dataset using GPT-4 Turbo. The training data encompasses two tasks: (a) Simple instruction re-caption: we adopt 8K descriptive image captions from LAION-GPT-4V and let GPT-4 inversely infer the corresponding user's short input and the target caption in the Stable Diffusion (SD) domain. (b) In-context prompt generation: based on a few high-quality real-world conversation contexts in LIMA and OpenAssistant2, we generate prompts that produce images suitable for the conversation context, bringing 5K instructions in total. For both kinds of data, in each query to GPT-4, we randomly sample 5 high-quality SD text-to-image prompts from GigaSheet as in-context examples to obtain target prompts for generation. We format our data to use <GEN> as a trigger to initiate the generation process and wrap the target caption within <h>...</h>. Following text generation, Mini-Gemini extracts target captions and utilizes SDXL to generate the corresponding image. Section 4. Experiments  In this section, we first outline our experimental framework, commencing with the experimental setup. Subsequently, we compare Mini-Gemini with leading methods on various benchmarks. Component-wise analysis and qualitative results are given at the end of this section. Subsection 4.1. Experimental Setup  Implementation Details. In this study, we instantiate Mini-Gemini with the CLIP-pretrained ViT-L for LR vision encoder and the LAION-pretrained ConvNext-L for HR vision encoder. For efficient training, we keep two vision encoders fixed and optimize the projectors of patch info mining in all stages. Meanwhile, we optimize the LLM during the instruction tuning stage only. Regarding the training scheme, we optimize all the models for 1 epoch with the AdamW optimizer and a Cosine learning schedule. In most cases, the initial learning rates for modality alignment and instruction tuning are respectively set at 1e-3 and 2e-5, with an adjusted rate of 1e-5 for the Mixtral-8x7B and Hermes-2-Yi-34B to ensure stable instruction tuning. The framework involves training on 8 A800 GPUs for standard machine configurations. For the largest model with Hermes-2-Yi-34B, we leverage 4 machines and complete the optimization within 2 days with DeepSpeed Zero3 strategy. For the HD version, the total cost is enlarged to about 4 days because of the extended visual tokens in LLMs. Datasets. For model optimization, we construct high-quality data for cross-modality understanding and generation. It mainly includes 1.2M caption pairs for modality alignment and 1.5M single- or multi-round conversations for instruction tuning. Moreover, we report results on widely-adopted zero-shot image-based benchmarks, including TextVQA, MMBench, MME, MM-Vet, MMMU, and MathVista datasets. Subsection 4.2. Main Results  Normal Resolution.