Browse and Concentrate: Comprehending Multimodal Content via Prior- LLM Context Fusion (2024)

Abstract: With the bloom of Large Language Models (LLMs), Multimodal Large Language Models (MLLMs) that incorporate LLMs with pre-trained vision models have recently demonstrated impressive performance across diverse vision-language tasks. However, they fall short to comprehend context involving multiple images. A primary reason for this shortcoming is that the visual features for each images are encoded individually by frozen encoders before feeding into the LLM backbone, lacking awareness of other images and the multimodal instructions. We term this issue as prior-LLM modality isolation and propose a two phase paradigm, browse-and-concentrate, to enable in-depth multimodal context fusion prior to feeding the features into LLMs. This paradigm initially “browses” through the inputs for essential insights, and then revisits the inputs to “concentrate” on crucial details, guided by these insights, to achieve a more comprehensive understanding of the multimodal inputs. Additionally, we develop training strategies specifically to enhance the understanding of multi-image inputs. Our method markedly boosts the performance on 7 multi-image scenarios, contributing to increments on average accuracy by 2.13% and 7.60% against strong MLLMs baselines with 3B and 11B LLMs, respectively.

## introduction multimodal large language models (mllms) have recently garnered attention for their surging popularity and impressive performance across diverse vision-language (vl) tasks @xcite @xcite . among these mllms, the paradigm that extending large language models (llms) with pre-trained vision encoders has shown remarkable abilities in visual reasoning and visual instruction-following @xcite what is the last three digits of the registration plate? 2023; @xcite . these models also draw attention for their feasibility and flexibility in adapting to varied scenarios and demands @xcite . despite its impressive abilities, this paradigm faces challenges that obscure a deeper understanding of multi-image and interleaved inputs @xcite @xcite . the approach of simply gluing up pre-trained vision and language models via intermediate components @xcite ) potentially neglects essential crossmodality and inter-image interactions, leading to the llm being presented with isolate visual and textual features without recognition of interleaved multimodal inputs. we refer to this problem as prior-llm modality isolation, which further divides two issues image-text isolation and interimage isolation. these challenges have received considerable attention but remain unresolved. firstly, image-text isolation happens when frozen vision encoders produce generic visual features, overlooking crucial target-specific information. for instance, in figure 1 (a) , the emphasis should be on the "registration plate". this plate, occupying only a minor area of the image, is prone to being overshadowed by predominant elements due to inadequate image-text interaction. to tackle this problem, @xcite and @xcite integrate textual instructions into visual feature extraction to enhance the responsiveness of these features to the given instructions. moreover, some researchers propose to alter the internal structure of llms to bridge the gap between visual and linguistic spaces @xcite . while these methods are effective in single-image scenarios, they do not address the concurrent fusion of multiple images. secondly, inter-image isolation arises from encoding images separately, disrupting semantic links among images and conveying misinformation of the multi-image context. this issue is particularly prevalent in scenarios involving interleaved and multiple images. as illustrated in figure 1 (b) , the moving direction regarding the other image should be considered. however, such relational information remains isolated and fails to transmit across images. consequently, the lack of awareness regarding relevant content from other images can lead to the exclusion of essential visual information. to handle this issue, recent studies have developed context schemes that aim to improve image-text correlations and the connections between multiple images @xcite . nevertheless, the prior-llm fusion of multimodal context are overlooked. to mitigate the two outlined issues, we utilize a cognitive strategy that mirrors the process through which humans typically understand new content: by first grasping the main ideas during an initial browsing and then revisiting the material to deepen their understanding with the browsing insights @xcite . inspired by this approach, we propose a novel paradigm named browse-and-concentrate (brote). this paradigm begins with a browsing phase to generate a condition context vector, serving as a collection of browsing insights, encapsulating the main intent and visual information derived from images. subsequently, a concentrating phase is employed to comprehend multimodal inputs, guided by the condition context vector. furthermore, to enhance the effectiveness of the browsing insights, we have developed training strategies that prompt the model to implicitly leverage these insights for more precise extraction of image features, allowing for the possibility of bypassing explicit browsing in some scenarios. our contributions can be summarized as follows: ## conclusion in this paper, we address the prior-llm modality isolation issue for both image-text and inter-image context, which lacks sufficient investigation in previous works. to mitigate this issue, we propose browse-and-concentrate paradigm that leverages the initial browsing insights for the prior-llm multimodal context fusion to stimulate more profound comprehending of multi-image and interleaved inputs. we present in-depth analysis on our proposed training strategies and the two approaches for implementing our proposed paradigm. the two approaches, explicitly or implicitly browse through and then concentrate on the context, exhibits comprehensive multimodal context understanding. our method demonstrates remarkable improvements on 7 multi-image tasks against strong baselines that enable prior-llm image-text fusion.