LLaVA also demonstrates impressive OCR (optical character recognition) ability, which is rarely covered in our training data. We hope these additional results and observations showcase the potential of LLaVA in various application areas. In future work, it is important to investigate these emergent behaviors more thoroughly and to understand the underlying mechanisms that enable LLaVA to demonstrate such generalization abilities. This will pave the way towards building better LMMs, including enhancing robustness, reducing biases, and improving the alignment and the scope of the learned vision-language representations. Training Details  We pre-train our model on the filtered CC-595K subset for 1 epoch with a learning rate of 2e-3 and a batch size of 128, and fine-tune on the proposed LLaVA-Instruct-158K dataset for 3 epochs, with a learning rate of 2e-5 and a batch size of 32. Following Vicuna, we use the Adam optimizer with no weight decay and a cosine learning rate with a warmup ratio of 3%. During finetuning, FSDP (Full Shard Data Parallel) and gradient checkpointing is used to save GPU memory, and offloading is not used. BF16 and TF32 are enabled to achieve a balance between speed and precision. We train all models with 8 A100s. Pretraining on CC-595K completes within 4 hours. Finetuning on Instruct-158K completes within 10 hours. Finetuning on ScienceQA completes within 4 hours. Data  Instructions for brief image description. The list of instructions used to briefly describe the image content present the same meaning with natural language variance. Instructions for detailed image description. The list of instructions used to describe the image content in detail present the same meaning with natural language variance. CC3M. We extract noun-phrases using Spacy for each caption over the whole CC3M dataset, and count the frequency of each unique noun-phrase. We skip noun-phrases whose frequency is smaller than 3, as they are usually rare combinations of concept and attributes that have already been covered by other captions. We then start from the noun-phrases with the lowest remaining frequency, and add the captions that contain this noun-phrase to the candidate pool. If the frequency of the noun-phrase is larger than 100, we randomly choose a subset of size 100 out of all its captions. This results in around 595K image-text pairs. The filtered dataset shows a good coverage of concepts whose frequency is higher than 3, but with a smaller number of image-text pairs. Prompts  The prompt used to generate image-based conversation from ChatGPT/GPT-4 is constructed using few-shot in-context-learning. The system message instructs the model to act as an AI visual assistant seeing a single image, described by five sentences. It is asked to design a diverse conversation about the photo, including questions about visual content like object types, counts, actions, and locations. It is also asked to include complex questions about background knowledge or events in the image, providing detailed, well-organized answers. The prompt emphasizes asking only questions that can be answered confidently from the image. Following the system message, a series of few-shot examples (user context and assistant response) are provided before the final user query.