Soft Knowledge Prompt: Help External Knowledge Become a Better Teacher to Instruct LLM in Knowledge-based VQA (2024)

Abstract: LLM has achieved impressive performance on multi-modal tasks, which have received ever-increasing research attention. Recent research focuses on improving prediction performance and reliability (e.g., addressing the hallucination problem). They often prepend relevant external knowledge to the input text as an extra prompt. However, these methods would be affected by the noise in the knowledge and the context length limitation of LLM. In our work, we focus on making better use of external knowledge and propose a method to actively extract valuable information in the knowledge to produce the latent vector as a soft prompt, which is then fused with the image embedding to form a knowledge-enhanced context to instruct LLM. The experimental results on knowledge-based VQA benchmarks show that the proposed method enjoys better utilization of external knowledge and helps the model achieve better performance.

## introduction although llm-based methods already perform well on many multi-modal tasks, knowledge-based vqa remains a challenging task, which requires outside knowledge beyond the image content to answer the question. in practice, llm often generates factually incorrect responses to the given questions since the knowledge in the llm model may be inaccurate, incomplete, and outdated. intuitively, a straightforward solution is to incorporate relevant external knowledge to assist llm in making a better prediction. in nature, such a paradigm is equivalent to using external knowledge as an additional prompt for llm. related research efforts mainly focus on training a better retriever to obtain relevant external knowledge @xcite , but the model passively receives the retrieved result. in fact, even the figure 1 : comparison between the traditional approach and our approach to utilize relevant external knowledge. giving the retrieved knowledge, the traditional approach directly prepends knowledge snippets to the input. in contrast, our approach dynamically selects and incorporates valuable information from the knowledge. as shown in figure 1 , while containing supporting content, the external knowledge snippet also contains distracting content. the precisely supporting content renders a solid foundation for a model to generate the correct output. various attempts have been made to filter retrieved snippets based on rules @xcite , which focus largely on the nlp tasks and neglect to excavate supporting information in a fine-grained manner. currently, existing models fail to pay sufficient attention to the supporting content and are prone to be distracted by surrounding sentences that share similar topics @xcite . multi-modal llm (mllm) has achieved tremendous success in multi-modal tasks. it leverages a projector to transform images into latent vectors that provide visual information context to the llm. given relevant external knowledge, prior approaches often directly prepend it with textual input to prompt llm. in fact, the textual knowledge snippets can be analogously transformed into latent vectors to prompt llm. such behavior offers the following advantages: 1) realize the finegrained extraction of pivotal information derived from external knowledge, facilitating the injection of knowledge into the prediction. 2) refine longer external knowledge texts into a small number of vectors, effectively avoiding adding numerous extra input tokens to llm, which has a context length limitation. inspired by the empirical evidence mentioned above, we propose a novel method (coined skp) to transform the external knowledge snippet into the latent vector as a soft knowledge prompt to better instruct llm for the knowledge-based vqa task. specifically, we extract the pivotal information in the external knowledge according to the visual and question information and transform it into latent vectors, where the attention mechanism is introduced to select the informative tokens derived from the external knowledge. then, knowledge latent vectors are fused with the visual embedding to form a knowledge-enhanced context for the question. furthermore, we propose a tailored training scheme to expedite the process of selecting and fusing pivotal knowledge. for implementation details, please refer to the code at https: //github.com/buaaw-ml/skp.git . the experimental results verify that the proposed method enjoys better utilization of external knowledge and helps the model achieve better performance. our main contribution can be summarized into threefold: ## method in the mllm framework, a projector module is often used to transform the image content into visual context vectors for llm. we focus on extracting pivotal information from relevant external knowledge and fusing it with the visual context vectors as a knowledge-enhanced context to prompt llm. the framework of the proposed method is illustrated in figure 2 . specifically, the input is well processed firstly (sec. 3.1), and we propose a method to mine valuable information from relevant external knowledge (sec. 3.2) and fuse it with the visual vectors to instruct llm (sec. 3.3). further, we adopt an ensemble-based approach to determine the final output (sec. 3.4). lastly, we design a tailored optimization scheme for the trainable components involved in our framework (sec. 3.5). ## conclusion this work proposes a method to extract the pivotal information from the relevant external knowledge, which is then fused with the visual information to form a knowledge-enhanced context. this approach protects llm from being misled by distracting information, and avoids adding extra input tokens to llm. the experimental results on knowledge-based vqa benchmarks demonstrate that the proposed method enjoys better utilization of external knowledge and is conducive to more accurate answers. in addition, compared with constructing a large and complex knowledge mining model, which requires lots of labeled training data and faces the problem of generalization, our method enjoys the characteristics of good interpretability, and simplicity of use. different from most recent works, we do not focus on constructing a better retriever to obtain more beneficial knowledge. given a relevant external knowledge, our work focuses on how to better utilize it to improve the prediction performance. we try our best to excavate its value, but the beneficial information within an external knowledge is limited. in future work, we will study how to combine our method and the retriever construction method to obtain a better performance improvement. in addition, we will also investigate the potential to improve the complementary between external knowledge and model implicit knowledge.