Tag-grounded Visual Instruction Tuning with Retrieval Augmentation (2024)

Abstract: Despite recent advances in the general visual instruction-following ability of Multimodal Large Language Models (MLLMs), they still struggle with critical problems when required to provide a precise and detailed response to a visual instruction: (1) failure to identify novel objects or entities, (2) mention of non-existent objects, and (3) neglect of object’s attributed details. Intuitive solutions include improving the size and quality of data or using larger foundation models. They show effectiveness in mitigating these issues, but at an expensive cost of collecting a vast amount of new data and introducing a significantly larger model. Standing at the intersection of these approaches, we examine the three object-oriented problems from the perspective of the image-to-text mapping process by the multimodal connector. In this paper, we first identify the limitations of multimodal connectors stemming from insufficient training data. Driven by this, we propose to enhance the mapping with retrieval-augmented tag tokens, which contain rich object-aware information such as object names and attributes. With our Tag-grounded visual instruction tuning with retrieval Augmentation (TUNA), we outperform baselines that share the same language model and training data on 12 benchmarks. Furthermore, we show the zero-shot capability of TUNA when provided with specific datastores.

## introduction multimodal large language models (mllm) have witnessed remarkable progress recently @xcite @xcite @xcite @xcite @xcite , exhibiting superior ability in following vision-andlanguage instructions. despite their effectiveness in providing general responses, their performance often degrade when required to give a detailed and accurate answer to the question associated with an image with novel objects, named entities or complex scenes with rich and subtle details. specifically, they frequently encounter challenges (fig. 1 ) in: 1. identifying novel objects and named entities, 2. preventing the generation of objects that do not align with the target images, and 3. delivering a comprehensive description that covers the details of the target images. we uncover the llm text embedding space clip embedding space span of llava data-1.2m span of datastore-15m span of llm space connector mapping retrieval mapping samples retrieved samples figure 2 : top: the process of translating image embeddings to text embeddings (llava @xcite ). bottom: image classification accuracy of clip @xcite and mllms built on it. despite the promising zero-shot capability of the vision encoder, such as clip @xcite , which is pre-trained with over 400m imagetext pairs, its generalizability is bottlenecked by the learnt mapping of the multimodal connector when integrated into the mllm framework. e.g., in the case of llava @xcite , the two-stage training data is significantly smaller compared to the pre-training data of its vision encoder clip (1.2m vs. 400m), as a result, the connector often fails to effectively map the out-of-distribution (ood) images to the corresponding llm text embeddings. therefore, llm fails to successfully identify image contents. mllms' degradation on image classification performance @xcite is a simple illustration. in fig. 2 (bottom), an obvious classification performance gap between mllms and their frozen vision encoder (clip) is observed. the absence of similar classification objects in llava's training data could be a critical factor, which makes it particular hard for the multimodal connector to translate ood clip embeddings of test images to llm text embeddings. one intuitive solution is to enrich the training datasets with more image-text pairs, however, as high-quality instruction-format data is particularly critical for visual instruction tuning @xcite , it is very expensive to build high-quality training data with hundreds of millions of imagetext pairs of varying quality. furthermore, the training could also become exceedingly burdensome. instead of directly improving the connector mapping with heavy training, could we build another lightweight new mapping as a complementary that effectively attends to objects, especially ood ones? motivated by retrieval augmented generation (rag) @xcite @xcite @xcite , we propose a retrieval mapping. as shown in fig. 2 (top), while the connector fails to correctly map the sample out of llava training data span to its corresponding text embedding in llm embedding space (i.e., the blue triangle sample is incorrectly mapped to the yellow square sample) , we introduce a largescale external datastore with a better coverage of novel objects, named entities, and attributes, for the retrieval of useful knowledge towards the input image. in this way, a new retrieval mapping could be built from the input image to corresponding llm text embeddings (green dashed line in fig. 2 ). while most existing works retrieve relevant captions as extra knowledge, it may not apply here because all three challenges mentioned above are oriented with object, where cleaner object-aware knowledge is urgent, instead of noisy captions. therefore, we want to retrieve tags of the images that are similar to the input image as extra knowledge, where we can further enrich each tag representation with image region feature and adaptive weights to fulfill the potential of useful tags. to this end, we introduce a tag-grounded visual instruction tuning with retrieval augmentation, termed tuna, that performs a knowledge-aware and taggrounded generation. with grounded tags, tuna is effective in identifying novel objects, named entities, and generate tag-oriented response which pays more attention to image details. we summarize our contributions as follows: (i) we identify potential factors hindering mllms and first propose a tag-grounded visual instruction tuning with retrieval-augmentation (tuna) with enhanced knowledge on novel objects, more atten-tion to details, and less mention of non-existent objects. (ii) to fulfill the potential of tags, we carefully designed the image-aware tag encoder, which produces tag embeddings enhanced by image features with an adaptive weight. (iii) we evaluate tuna on extensive benchmarks along with a series of qualitative results, and show its zero-shot capability when provided with specific datastores. ## related works multimodal large language models. mllms evolve rapidly nowadays. with llms, while existing works @xcite ) enable basic visual tasks like visual question answering, more recent works @xcite shows proficiency in image-text dialogues through alignment and fine-tuning. subsequent research @xcite @xcite @xcite @xcite enhances llms by emphasizing data quality and diversity. with grounding data, a branch of works @xcite @xcite improves llms' grounding capability. despite their evolution, as they share a similar multimodal connector module that performs image-totext translation, a lingering fundamental problem persists: out-of-distribution (ood) images, such as novel objects, named entities, new scenes, etc., cannot be translated to text embeddings effectively, leading to misaligned answers, missing details or mention of non-existent objects from llm. ## tag-ground visual instruction tuning in this section, we first introduce how we extract tags from 15m captions from cc12m @xcite and cc3m @xcite . then we present how we build and use the datastore, followed by the illustration of tuna. ## experiment in this section, we first present the training details of tuna and benchmarks. then we introduce quantitative and qualitative comparison with popular open-source models, followed by detailed analysis experiments and ablation studies. training details. tuna is finetuned on instruction data for one epoch, following existing works @xcite . we consider two different instruction-following datasets in our experiments: llava-665k @xcite and sharegpt4v-665k (chen et al., 2023c) as our instruction-following data during fine-tuning separately, resulting in two versions of our model, tuna and tuna + . sharegpt4v-665k contains instruction-following data with higher quality. details on datasets are available in appendix c. we apply a learning rate of 2e-5 and a batch size of 128. the training takes 12∼14 hours with 8 a100 gpus with zero3. details are available in appendix c. benchmarks. we compare tuna with baselines on 12 benchmarks, including vqa benchmarks and multimodal benchmarks designed for llms. details are available in appendix g. 4.1 comparison with baselines main results. in tab. 2, we provide a quantitative comparison of tuna with popular open-source mllms. on 12 benchmarks, tuna consistently outperforms previous llms that are finetuned from the same instruction-tuning datasets as ours with the same configuration on the vision encoder and language model (vicuna-7b), especially on recent multimodal benchmarks with more notable im-provements. as the size of llm and different choices of instruction-following data can significantly improve the model performance, we mark the models gray that are equipped with a larger 13b language model or finetuned from currently unavailable datasets of higher quality and quantity. specifically, llava-1.6 (or llava-next) foot_0 is finetuned from larger instruction-following data of higher quality, with additional user instruct data. besides, it equips the better vision encoder with dynamic high resolution, known as anyres (ar). although it is not a fair comparison, we still outperform llava-1.6 in mmb cn , mmb and pope, and the corresponding 13b models in mmb cn , mmb, pope and llava-w. how can tuna improve the recognition of novel objects and entities? as visualized in fig. 2 (top), with our 15m large-scale datastore, the new retrieval mapping could greatly compensate for the original llava multimodal connector that learns from around 1m data. with the additional mappings from retrieval data, tuna is expected to show particularly improvements over questions towards novel objects or entities in the given input image. we show sub-tasks from mme @xcite and mmb @xcite that consists of such questions in tab. 3. we gain obvious improvements over the baseline in most sub-tasks. we also show several vqa examples from multimodal benchmarks in fig 1 and fig 5 . in fig 1 , all of the baselines fail to correctly identify this fruit as a mangosteen, including llava-1.5. it is reasonable as mangosteens do not appear in the its training data, which makes it particularly hard for the connector to map it to somewhere close to text embeddings of "mangosteen" in the llm embedding space, as illustrated in fig 2 . when the question about the given image is a little tricky, e.g., in fig 5 (a), the mllm is asked if a painting of a building exists in the form of architecture, llava-1.5 is confused on whether it is a real architecture or a painting. however, tuna easily distinguished it from real architectures with additional knowledge from retrieved tags of similar images in datastore. how can tuna help to identify the existence of objects? with an input image, the retrieved images are often similar to it or in the similar context. intuitively, the retrieved images are very likely to contain similar elements or objects to the input image. therefore, the tags could be helpful to provide additional hints to the llm to pay special attention to them about their existence. we evaluate our model on pope @xcite , a benchmark designed towards the existence of objects. results are available in tab. 4, we outperform competing baselines including referring and grounding mllms such as ferret @xcite and shikra @xcite . a simple example is shown in fig 5 @xcite . when the object is visually imperceptible in the image, hints from tags are very helpful. how can tuna attend to rich details with tags? while most of the traditional vqa bench-what is the name of this famous sight? please describe this photo in detail. llava: the image features a breathtaking view of the diamond head crater, a rocky island situated in the middle of a body of water, possibly an ocean. the island is surrounded by a beautiful beach with a sandy shore, and it appears to be a popular tourist destination. in the distance, there are several boats scattered around the water, likely providing transportation or leisure activities for visitors. the scene is further enhanced by the presence of a few people enjoying the beach, with some of them closer to the water's edge and others further back on the shore. the combination of the rocky island, the beach, and the boats creates a picturesque and serene atmosphere. ours: the image presents an aerial view of diamond head, a jagged island with a coastline that stretches along the water. the island is covered in green vegetation, and the water appears to be a bright blue. the coastline is lined with cliffs, giving the island a dramatic appearance. in the background, there are several houses and buildings scattered across the island, indicating human presence and development. the combination of the island's natural beauty and the presence of human structures creates a captivating scene. imprecise low-quality answers are marked in red and high-quality parts are marked in green. tuna does not mention nonexistent objects and gives a more detailed description. ## conclusion in this paper, we discussed three challenges for mllms: (1) mention of non-existent objects, (2) neglect of visual details and (3) failure to identify novel objects and entities, and one of the potential causes: the bottleneck from the image-totext translation. to alleviate these problems, we introduced tuna, a tag-grounded visual instruction tuning framework with retrieval-augmentation, which achieves competing performance over 12 vqa and multimodal benchmarks, compared to baselines with the same llm and finetuning data.