Cognitive Visual-Language Mapper: Advancing Multimodal Comprehension with Enhanced Visual Knowledge Alignment (2024)

Abstract: Evaluating and Rethinking the current landscape of Large Multimodal Models (LMMs), we observe that widely-used visual-language projection approaches (e.g., Q-former or MLP) focus on the alignment of image-text descriptions yet ignore the visual knowledge-dimension alignment, i.e., connecting visuals to their relevant knowledge. Visual knowledge plays a significant role in analyzing, inferring, and interpreting information from visuals, helping improve the accuracy of answers to knowledge-based visual questions. In this paper, we mainly explore improving LMMs with visual-language knowledge alignment, especially aimed at challenging knowledge-based visual question answering (VQA). To this end, we present a Cognitive Visual-Language Mapper (CVLM), which contains a pretrained Visual Knowledge Aligner (VKA) and a Fine-grained Knowledge Adapter (FKA) used in the multimodal instruction tuning stage. Specifically, we design the VKA based on the interaction between a small language model and a visual encoder, training it on collected image-knowledge pairs to achieve visual knowledge acquisition and projection. FKA is employed to distill the fine-grained visual knowledge of an image and inject it into Large Language Models (LLMs). We conduct extensive experiments on knowledge-based VQA benchmarks and experimental results show that CVLM significantly improves the performance of LMMs on knowledge-based VQA (average gain by 5.0%). Ablation studies also verify the effectiveness of VKA and FKA, respectively.

## introduction recent large multimodal models (lmms) such as gpt-4v (openai, 2023), gemini @xcite , 2023), minigpt-4 @xcite , instruct-blip @xcite , llava @xcite , and many others, have achieved impressive performance in a variety of visual understanding and reasoning tasks, especially on visual question answering (vqa) @xcite . current opensource lmms are usually constructed by combining pertained visual encoders and large language models (llms), as depicted in the bottom part of figure 1 , where a visual mapping network (e.g., q-former @xcite , linear @xcite , or mlp @xcite ) is employed to project visual representations into the language space of llms. although such lmms have achieved powerful visual understanding capability similar to gpt-4v and genimi on some image understanding tasks such as image captioning @xcite , visual dialogue @xcite , visual entailment @xcite , and vqa @xcite , they often fall short of knowledge-based vqa, which necessitates relevant knowledge to answer these visual questions. as the cases illustrated in figure 1 , these advanced lmms (including gpt-4v and gemini-pro) can not give correct answers to simple visual information seeking questions: who is the manufacturer of this aircraft; what country does this building belong to?. in light of this, rethinking the construction process of lmms @xcite @xcite @xcite from the initial pretraining stages, we discover that these visual mapping networks trained on massive image-text captioning pairs simply transfer visual features to their language descriptions. they overlook the visual language knowledge-dimension alignment. i.e., connecting visuals to their relevant knowledge. as we know, visual knowledge @xcite plays a pivotal role in the way humans understand and interact with the world. it extends beyond the mere ability to recognize and interpret visuals, incorporating an understanding of spatial relationships, patterns, and symbols, which are essential components of human cognition @xcite . additionally, previous works also demonstrated that introducing visual knowledge @xcite @xcite can improve the performance of pretrained language models on natural language understanding @xcite and open-ended text generation tasks @xcite . inspired by these insights, we focus on enhancing lmms through the introduction of visual-language knowledge alignment, going beyond the conventional scope of visual-language integration. to this end, we present a cognitive visual-language mapper (cvlm), which contains a pretrained visual knowledge aligner (vka) and a fine-grained knowledge adapter (fka). specifically, we devise vka based on a small language model that interacts with fine-grained image representation in each block. the output hidden states of vka are fed into the llm as the knowledge embedding tokens by a linear projection layer. to make vka effectively capture image-relevant knowledge, we first train it on image-knowledge pairs @xcite collected from wikipedia 2 via the next tokens prediction. like q-former @xcite and prefixtuning @xcite , we only fine-tune some learnable query tokens and the linear layer to acquire fixed-length visual knowledge representa-2 https://en.wikipedia.org/wiki/wikipedia:images tion and convert it into the representation space of llm. in addition, considering that visual objects contain fine-grained visual knowledge, we introduce fka to gain comprehensive visual knowledge of an image and distill valuable visual knowledge from the whole knowledge representation sequence. the output knowledge vectors of fka are injected into each layer of llms to realize in-depth interactions between llms and detailed visual knowledge. by doing so, cvlm is capable of connecting visuals to relevant knowledge, enabling lmms to utilize them during multimodal understanding and generation. to verify the effectiveness of cvlm, we conduct extensive experiments on image-centered, knowledge-based, and complex visual reasoning question-answering scenarios: vqav2 @xcite , okvqa @xcite , a-okvqa @xcite , infoseek @xcite , textvqa @xcite , and seedbench @xcite . the experimental results show that cvlm significantly outperforms previous strong baselines such as llava-v1.5. the ablation and case studies indicate that cvlm is capable of linking visual knowledge and improving performance on knowledge-intensive tasks via the introduced aligner and adapter. our contributions can be summarized as follows: ## conclusion in this work, we introduce the cognitive visual-language mapper (cvlm), an innovative approach that goes beyond the conventional alignment of visual and textual descriptions by incorporating visual-knowledge alignment. specifically, we have developed a visual knowledge aligner (vka) that facilitates the projection of visual knowledge by acting as a bridge between the vision encoder and llm. additionally, we have inte-grated a fine-grained visual knowledge adapter (fka) during the multimodal instruction tuning stage, which is designed to distill more precise knowledge pertinent to images and instructions. our experimental findings demonstrate that cvlm outperforms several prominent lmms that lack visual knowledge alignment. our ablation studies highlight the effectiveness of vka and fka.