G rounding GPT : Language Enhanced Multi-modal Grounding Model (2024)

Abstract: Multi-modal large language models (MLLMs) have demonstrated remarkable performance across various tasks. However, these models often prioritize capturing global information and overlook the importance of perceiving local information. This limitation hinders their ability to effectively understand fine-grained details and handle grounding tasks that necessitate nuanced comprehension. Although some recent works have made strides in this, they have primarily focused on single-modality inputs. Therefore, we propose GroundingGPT , an end-to-end language enhanced multi-modal grounding model. It is designed to perform fine-grained grounding tasks for three modalities: image, video and audio. To enhance the model’s performance, we adopt a coarse-to-fine training strategy, utilizing a three-stage training approach to progressively enhance the model’s semantic awareness and fine-grained understanding capabilities. Additionally, we employ a diversified stage-specific dataset construction pipeline, developing a multi-modal, multi-granularity dataset tailored for training the model in different stages. Extensive experiments conducted on multiple multi-modal benchmarks demonstrate that our model achieves impressive fine-grained understanding of multi-modal inputs on grounding tasks while maintaining or improving its global comprehension capabilities. Our code, model, and dataset are available at https://github.com/lzw-lzw/GroundingGPT.

## introduction building upon the capabilities of large language models (llms), research on multi-modal large language models (mllms) has also advanced, enabling understanding across a broader range of modalities. representative models such as llava @xcite and minigpt-4 @xcite align visual features obtained from image encoders with llm embedding space through visual instruction tuning, facilitating tasks such as image captioning and visual question answering. however, existing mllms primarily focus on capturing global information while neglecting the fine-grained local information in multimodal inputs. this limitation restricts their applicability in grounding tasks requiring a more detailed understanding. shikra @xcite , bubogpt @xcite and ferret @xcite have explored techniques that enable finer alignment and understanding of inputs. by considering local-level information, these models exhibit enhanced performance in grounding or referring tasks. these methods provide insights into finegrained understanding, but they are primarily limited to a single modality. there is still significant potential for exploring fine-grained understanding across other modalities. to address the aforementioned issue, this paper proposes groundinggpt, a language enhanced multi-modal grounding model, which is an endto-end unified large language model designed to perform multi-modal grounding and understanding tasks across various modalities, including image, video, and audio. the comparison between our model and other models can be found in to incorporate spatial and temporal information, we directly represent coordinates and timestamps as textual numbers, eliminating the need for vocabulary expansion. for training groundinggpt, we design a three-stage coarse-to-fine training strategy. in the first stage, we align each pre-trained multimodal encoder with the llm embedding space using modality-specific adapters. in the second stage, we aim to enable the model to capture finegrained information, including coordinates and timestamps. in the third stage, we perform multigranularity instruction tuning to refine the model's models image grounding video grounding audio grounding multi-turn dialog e2e llava video-llama shikra ferret bubogpt llava-grounding groundinggpt to summarize, our contributions are as follows: ## methods we introduce the overall architecture of the ground-inggpt model in this section. additionally, we will present our three-stage coarse-to-fine training strategy and data construction pipeline. ## conclusion in this paper, we introduce groundinggpt, a unified end-to-end multi-modal grounding model. to the best of our knowledge, this is the first multimodal large language model capable of performing multi-modal grounding and understanding tasks. we adopt a three-stage coarse-to-fine training strategy, accompanied by the construction of stagespecific training datasets, to effectively train the model. our model demonstrates remarkable perfor-mance in multi-modal grounding and understanding tasks. extensive experiments conducted on a wide range of mllm benchmarks confirm the effectiveness and generality of our model. to foster further advancements in this field, we make our model, code, and dataset openly accessible.