C o LL a VO : Crayon Large Language and Vision m O del (2024)

Abstract: The remarkable success of Large Language Models (LLMs) and instruction tuning drives the evolution of Vision Language Models (VLMs) towards a versatile general-purpose model. Yet, it remains unexplored whether current VLMs genuinely possess quality object-level image understanding capabilities determined from ‘what objects are in the image?’ or ‘which object corresponds to a specified bounding box?’. Our findings reveal that the image understanding capabilities of current VLMs are strongly correlated with their zero-shot performance on vision language (VL) tasks. This suggests that prioritizing basic image understanding is crucial for VLMs to excel at VL tasks. To enhance object-level image understanding, we propose Crayon Large Language and Vision mOdel (CoLLaVO), which incorporates instruction tuning with Crayon Prompt as a new visual prompt tuning scheme based on panoptic color maps. Furthermore, we present a learning strategy of Dual QLoRA to preserve object-level image understanding without forgetting it during visual instruction tuning, thereby achieving a significant leap in numerous VL benchmarks in a zero-shot setting.

## introduction spurred by the enduring ambition for artificial general intelligence (agi) and the success of language models such as bert @xcite , gpt-3 @xcite , and llama @xcite , there has been a surge in demand for a general-purpose model in a task-unified format via natural language instruction, leading to the emergence of instruction tuning @xcite . building on the success of large language models (llms) and instruction tuning, instructblip @xcite , figure 1 : zero-shot performance of collavo-7b on challenging vl datasets compared with closed-source vlms @xcite @xcite . note: the scores of mme are rescaled by 1/20 to match the scales with the accuracies of others. llava1.5 @xcite , and qwen-vl @xcite have either directly designed or utilized visual instruction tuning datasets for a wide range of vision language (vl) tasks using natural language instructions. consequently, they have become paradigm-shifting in vision language models (vlms), showcasing remarkable zero-shot performance in vl tasks. however, it is yet uncharted whether the current leading vlms truly possess a comprehensive understanding of fine-grained object information, and how this understanding influences their zero-shot performance in vl tasks related to each object. hence, we delve into the analysis of object-level image understanding and zero-shot performance in vl tasks across different objects. to illustrate the behavior of object-level image understanding, we employ four strong baselines: blip2 @xcite , instructblip @xcite , llava1.5 @xcite , and qwen-vl @xcite . we pose two types of simple questions to gauge their object-level understanding such as: (1) 'is there any {object name} in this image?' (class2binary: c2b), and (2) 'which object is in the specified bounding box [x min , y min , x max , y max ]?' (box2class: b2c). we then evaluate the is there an instance of the category "bottle" in the image? there may be multiple instances which are grouped together. answer the question using either yes or no. blip2: no gt: yes is there an instance of the category "hair drier" in the image? there may be multiple instances which are grouped together. answer the question using either yes or no. is there an instance of the category "handbag" in the image? there may be multiple instances which are grouped together. answer the question using either yes or no. ## research backgrounds visual prompting. researchers have prioritized enhancing natural language prompts in constructing instruction tuning datasets for llms @xcite @xcite . on the other hand, dealing with vlms offers new opportunities to manipulate both visual and textual aspects of prompts. earlier studies on visual prompting have focused on techniques such as learnable token embedding concatenated with visual embedding @xcite , or learned perturbation patterns directly applied to an input image @xcite @xcite . while these methods aim to find the optimal visual prompt, the learned visual prompts lack human interpretability, hindering the understanding of their effectiveness. to address this, current vlms use humaninterpretable visual prompts such as marks @xcite @xcite or semantic masks @xcite . shtedritski et al. (2023) draw red circles on images and then demonstrate that clip @xcite , by itself, can recognize the simple visual prompts on images, showing improved zero-shot performance for tasks such as referring expressions comprehension and key point localization. by using seem @xcite or sam (kirillov in brief, previous studies have focused on guiding vlms towards specific areas using marks and semantic masks. similar to @xcite , we propose crayon prompt encompassing all foreground and background objects at once. however, compared with a direct visual prompt on the image @xcite @xcite @xcite , the crayon prompt is injected into image embedding features at every transformer @xcite layer in a backbone mlm to keep the image intact and not disrupt its raw visual context. the crayon prompt provides semantic information about objects in the image and their numbering, similar to how positional embedding @xcite provides sequential information about the relative orders of token embedding features. ## collavo model architecture and prompt protocol. the structure of collavo, as illustrated in figure 4 , comprises a vision encoder, crayon prompt, a backbone mlm, and mlp connectors between the vision and language components. clip @xcite is considered as the vision encoder, benefiting from its adeptness in image understanding. the mlm utilized in collavo is from internlm-7b @xcite , which is a multilingual foundation model instruction tuned by 1.6t multilingual datasets with rlhf @xcite @xcite . moreover, two fully-connected mlps with gelu activa- ## discussion and conclusion we have shown the effectiveness of a coco classes for panoptic color map @xcite , ferret @xcite , vistalllm @xcite , qwen-vl @xcite , cogvlm-grounding @xcite , and collavo on several object grounding benchmarks: refcoco, refcoco+, and refcocog @xcite . even though collavo did not use object grounding dataset (refcoco) in training phase, collavo shows comparable zero-shot object grounding performances, compared with (no zero-shot) other models specifically targeting object grounding task trained with refcoco grounding dataset.