Section 2. Related Work  Multimodal Instruction-following Agents. In computer vision, existing works that build instruction-following agents can be broadly categorized into two classes: First, end-to-end trained models, which are separately explored for each specific research topic. For example, the vision-language navigation task and Habitat require the embodied AI agent to follow natural language instructions and take a sequence of actions to complete goals in visual environments. In the image editing domain, given an input image and a written instruction that tells the agent what to do, InstructPix2Pix edits images by following the human instructions. Second, a system that coordinates various models via LangChain or LLMs, such as Visual ChatGPT, X-GPT, MM-REACT, VisProg, and ViperGPT. While sharing the same goal in building instruction-following agents, we focus on developing an end-to-end trained language-vision multimodal model for multiple tasks. Instruction Tuning. In the natural language processing (NLP) community, to enable LLMs such as GPT-3, T5, PaLM, and OPT to follow natural language instructions and complete real-world tasks, researchers have explored methods for LLM instruction-tuning, leading to instruction-tuned counterparts such as InstructGPT/ChatGPT, FLAN-T5, FLAN-PaLM, and OPT-IML, respectively. It turns out that this simple approach can effectively improve the zero- and few-shot generalization abilities of LLMs. It is thus natural to borrow the idea from NLP to computer vision. More broadly, the teacher-student distillation ideas with foundation models have been studied in other topics such as image classification. Flamingo can be viewed as the GPT-3 moment in the multimodal domain, due to its strong performance on zero-shot task transfer and in-context-learning. Other LMMs trained on image-text pairs include BLIP-2, FROMAGe, and KOSMOS-1. PaLM-E is an LMM for embodied AI. Based on the recent "best" open-source LLM LLaMA, OpenFlamingo and LLaMA-Adapter are open-source efforts that enable LLaMA to use image inputs, paving the way to build open-source multimodal LLMs. While these models present promising task transfer generalization performance, they are not explicitly tuned with vision-language instruction data, and their performance in multimodal tasks usually falls short compared to language-only tasks. In this paper, we aim to fill this gap and study its effectiveness. Finally, note that visual instruction tuning is different from visual prompt tuning: the former aims to improve the model's instruction-following abilities, while the latter aims to improve the parameter-efficiency in model adaptation. Section 3. GPT-assisted Visual Instruction Data Generation  The community has witnessed a surge in the amount of public multimodal data such as image-text pairs, ranging from CC to LAION. However, when it comes to multimodal instruction-following data, the available amount is limited, partially because the process for creating such data is time-consuming and less well-defined when human crowd-scouring is considered. Inspired by the success of recent GPT models in text-annotation tasks, we propose to leverage ChatGPT/GPT-4 for multimodal instruction-following data collection, based on the widely existing image-pair data. For an image X_v and its associated caption X_c, it is natural to create a set of questions X_q with the intent to instruct the assistant to describe the image content. We prompt GPT-4 to curate such a list of questions. Therefore, a simple way to expand an image-text pair to its instruction-following version is: Human: X_q X_v <STOP> Assistant: X_c <STOP>. Though cheap to construct, this simple expanded version lacks diversity and in-depth reasoning in both the instructions and responses. To mitigate this issue, we leverage language-only GPT-4 or ChatGPT as the strong teacher (both accept only text as input), to create instruction-following data involving visual content. Specifically, in order to encode an image into its visual features to prompt a text-only GPT, we use two types of symbolic representations: First, Captions typically describe the visual scene from various perspectives. Second, Bounding boxes usually localize the objects in the scene, and each box encodes the object concept and its spatial location. This symbolic representation allows us to encode the image as an LLM-recognizable sequence. We use COCO images and generate three types of instruction-following data. For each type, we first manually design a few examples. They are the only human annotations we have during data collection, and are used as seed examples in in-context-learning to query GPT-4. The first type is Conversation. We design a conversation between the assistant and a person asking questions about this photo. The answers are in a tone as if the assistant is seeing the image and answering the question. A diverse set of questions are asked about the visual content of the image, including the object types, counting the objects, object actions, object locations, and relative positions between objects. Only questions that have definite answers are considered. The second type is Detailed description. To include a rich and comprehensive description for an image, we create a list of questions with such an intent. We prompt GPT-4 then curate the list. For each image, we randomly sample one question from the list to ask GPT-4 to generate the detailed description. The third type is Complex reasoning. The above two types focus on the visual content itself, based on which we further create in-depth reasoning questions. The answers typically require a step-by-step reasoning process by following rigorous logic. We collect 158K unique language-image instruction-following samples in total, including 58K in conversations, 23K in detailed description, and 77k in complex reasoning, respectively. We ablated the use of ChatGPT and GPT-4 in our early experiments, and found that GPT-4 consistently provides higher quality instruction-following data, such as spatial reasoning. Section 4. Visual Instruction Tuning  Subsection 4.1. Architecture  The primary goal is to effectively leverage the capabilities of both the pre-trained LLM and visual model. We choose Vicuna as our LLM, as it has the best instruction following capabilities in language tasks among publicly available checkpoints. For an input image X_v, we consider the pre-trained CLIP visual encoder ViT-L/14, which provides the visual feature Z_v equals g of X_v. The grid features before and after the last Transformer layer are considered in our experiments. We consider a simple linear layer to connect image features into the word embedding space. Specifically, we apply a trainable projection matrix W to convert Z_v into language embedding tokens H_v, which have the same dimensionality as the word embedding space in the language model. The equation is H_v equals W dot Z_v. Thus, we have a sequence of visual tokens H_v. Note that our simple projection scheme is lightweight, which allows us to iterate data centric experiments quickly. More sophisticated schemes to connect the image and language representations can also be considered, such as gated cross-attention in Flamingo and Q-former in BLIP-2. We leave exploring possibly more effective and sophisticated architecture designs for LLaVA as future work. Subsection 4.2. Training  For each image X_v, we generate multi-turn conversation data. We organize them as a sequence, by treating all answers as the assistant's response. The instruction at the t-th turn, X_instruct^t, is defined as follows: for the first turn (t=1), we randomly choose between [X_q^1, X_v] or [X_v, X_q^1]. For all remaining turns (t>1), the instruction is simply X_q^t. This leads to a unified format for the multimodal instruction-following sequence. We perform instruction-tuning of the LLM on the prediction tokens, using its original auto-regressive training objective. Specifically, for a sequence of length L, we compute the probability of the target answers X_a. This probability, p(X_a | X_v, X_instruct), is the product from i equals 1 to L of the probability of each token x_i, conditioned on the image X_v, and all preceding instruction and answer tokens. Here, theta represents the trainable parameters. For readability, we note that the image is grounded for all answers, and system messages and stop tokens are omitted from the notation. For LLaVA model training, we consider a two-stage instruction-tuning procedure. Stage 1: Pre-training for Feature Alignment. To strike a balance between concept coverage and training efficiency, we filter CC3M to 595K image-text pairs. These pairs are converted to the instruction-following data using the naive expansion method described earlier. Each sample can be treated as a single-turn conversation. To construct the input instruction, for an image X_v, a question X_q is randomly sampled, which is a language instruction to request the assistant to describe the image briefly. The ground-truth prediction answer X_a is the original caption. In training, we keep both the visual encoder and LLM weights frozen, and maximize the likelihood with trainable parameters being only the projection matrix W. In this way, the image features H_v can be aligned with the pre-trained LLM word embedding. This stage can be understood as training a compatible visual tokenizer for the frozen LLM. Stage 2: Fine-tuning End-to-End. We always keep the visual encoder weights frozen, and continue to update both the pre-trained weights of the projection layer and LLM in LLaVA; i.e., the trainable parameters are W and the LLM parameters phi. We consider two specific use case scenarios:  First, Multimodal Chatbot. We develop a Chatbot by fine-tuning on the 158K language-image instruction-following data. Among the three types of responses, conversation is multi-turn while the other two are single-turn. They are uniformly sampled in training. Second, Science QA. We study our method on the ScienceQA benchmark, the first large-scale multimodal science question dataset that annotates the answers with detailed lectures and explanations. Each question is provided a context in the form of natural language or an image. The assistant provides the reasoning process in natural language and selects the answer among multiple choices. For training, we organize the data as a single turn conversation, the question and context as the instruction, and reasoning and answer as the response. Section 5. Experiments  We assess the performance of LLaVA in instruction-following and visual reasoning capabilities with two primary experimental settings: multimodal chatbot and the ScienceQA dataset, respectively. We train all models with 8 A100s, following Vicuna's hyperparameters. We pre-train our model on the filtered CC-595K subset for 1 epoch with a learning rate of 2e-3 and a batch size of 128, and fine-tune on the proposed LLaVA-Instruct-158K dataset for 3 epochs, with a learning rate of 2e-5 and a batch size of 32. Multimodal Chatbot  We developed a chatbot demo to show the image understanding and conversation abilities of LLaVA, and to study how well LLaVA is able to digest visual inputs and exhibit instruction-following capabilities. We first use the examples in the original GPT-4 paper. For comparisons, we quote the prompt and response of the multimodal GPT-4 from their paper, and query BLIP-2 and OpenFlamingo model checkpoints to get their response. Surprisingly, although LLaVA is trained with a small multimodal instruction-following dataset (around 80K unique images), it demonstrates quite similar reasoning results with multimodal GPT-4 on these examples. Note that while these images are out-of-domain for LLaVA, LLaVA is still able to understand the scenes and follow the question instruction to provide a reasonable response. In contrast, BLIP-2 and OpenFlamingo focus on describing the image, instead of following the user instruction to answer in an appropriate manner. Quantitative Evaluation. To gain a systematic understanding of the performance of LLaVA, we propose a quantitative metric to measure the model's instruction-following capability on multimodal data. Inspired by previous work, we leverage GPT-4 to measure the quality of generated responses. Specifically, we create triplets consisting of image, ground-truth textual descriptions, and question. The candidate models (e.g., LLaVA) predict the answers based on the question and the image. To provide an approximate theoretical upper bound, we create a reference prediction based on the question and the ground-truth textual descriptions, using the text-only GPT-4.