P anda GPT : One Model To Instruction-Follow Them All (2023)

Abstract: We present PandaGPT, an approach to emPower large lANguage moDels with visual and Auditory instruction-following capabilities. Our pilot experiments show that PandaGPT can perform complex tasks such as detailed image description generation, writing stories inspired by videos, and answering questions about audios. More interestingly, PandaGPT can take multimodal inputs simultaneously and compose their semantics naturally. For example, PandaGPT can connect how objects look in an image/video and how they sound in an audio. To do so, PandaGPT combines the multimodal encoders from ImageBind and the large language models from Vicuna. Notably, only aligned image-text pairs are required for the training of PandaGPT. Thanks to the strong capability of ImageBind in embedding data from different modalities into the same space, PandaGPT displays emergent, i.e. zero-shot, cross-modal behaviors for data other than image and text (e.g., video, audio, depth, thermal, and IMU). We hope that PandaGPT serves as an initial step toward building AGI that can perceive and understand inputs in different modalities holistically, as we humans do.

## introduction humans possess remarkable abilities to perceive and understand information from diverse sensory modalities, such as seeing a painting and hearing an audio guide. analogously, to learn simultaneously, holistically, and directly from many different forms of information holds great promise for enabling machines to have a more comprehensive and better understanding of the world. to this end, there has been an emergent interest in developing artificial intelligence (ai) systems capable of perceiving and understanding information from multiple modalities simultaneously in a manner similar to humans. however, much of the prior research has focused on tackling individual modalities in isolation. for instance, while significant progress has been made in text-to-image retrieval and generation @xcite , visually-grounded instruction following @xcite , and speech understanding and generation @xcite , these advances have largely been confined to separate combinations of text and other modalities or, at best, a few visual modalities (e.g., image and video). these models are limited in their ability to connect information from different modalities and lack the capacity to perceive and understand multimodal inputs holistically, thereby neglecting the inherent richness and complementary nature of multimodal data. in this paper, we present pandagpt, the first general-purpose model capable of instructionfollowing data from six modalities. pandagpt leverages the power of multimodal encoders from imagebind @xcite and the expressive language models from vicuna @xcite , demonstrating impressive and emergent cross-modal capabilities across six modalities, namely, image/video, text, audio, depth, thermal, and inertial measurement units (imu). crucially, pandagpt achieves these capabilities despite being only trained on aligned image-text pairs, thanks to the shared embedding space provided by image-bind. this integration of multimodal information enables pandagpt to perform a wide range of tasks, including generating detailed descriptions of images, composing engaging stories inspired by videos, and providing accurate answers to questions about audio inputs. most interestingly, the core innovation of pandagpt lies in its ability to naturally compose the semantics of multimodal inputs, which enables a rich set of compositional multimodal tasks across different modalities. for example, it can seamlessly connect the visual appearance of objects in a photo with their corresponding sounds in an audio clip, producing a cohesive and comprehensive understanding of the scene. these cross-modal capabilities empower the model to go beyond traditional unimodal analysis. we hope pandagpt serves as an initial step toward building agi that can perceive and understand inputs in different modalities holistically, as humans do. multi-modal alignment. feature alignment among multiple modalities has attracted great interest for its applications such as cross-modal retrieval @xcite @xcite . recently, clip @xcite learns a joint embedding space for image and text. flamingo @xcite , blip-2 @xcite , and magic @xcite bridge powerful pre-trained vision-only and language-only models and show strong zero-shot abilities. audio-clip @xcite adds audio into the clip framework for audio classification. image-bind @xcite learn a joint embedding across six different modalities (image/video, text, audio, depth, thermal, and imu data) using image-paired data only. more recently, there has been a surge of interest to combine multi-modal alignment and large language models for multimodal instruction following. llava @xcite , mini-gpt4 @xcite , and video-llama @xcite enable visuallygrounded instruction following. detgpt @xcite proposes reasoning-based object detection. speechgpt @xcite adds speech understanding and generation abilities to llms. however, these advances have largely been confined to separate combinations of text and other modalities (e.g., image/video or audio). ## method pandagpt combines the multi-modal encoders from imagebind @xcite and the large language models from vicuna @xcite , achieving impressive capabilities in vision-and audio-grounded instruction following tasks. to align the feature space of multimodal encoders from imagebind and large language models from vicuna foot_0 , we train pandagpt using 160k imagelanguage instruction-following data released by liu et al. (2023) and zhu et al. (2023). each training instance consists of an image i and a multi-turn conversation data (x 1 , y 1 , ..., x n , y n ), where x i and y i are the human's instruction and the system's response at the i-th turn, respectively. to reduce the number of trainable parameters, we only train (i) a linear projection matrix f to connect the representation produced by imagebind to vicuna; and (ii) additional lora (hu et al., 2021) weights on the vicuna's attention modules. foot_1 figure 1 illustrates the architecture of pandagpt. ## capabilities of pandagpt compared to existing multimodal instructionfollowing models trained individually for one particular modality, pandagpt can understand and combine the information in different forms together, including image/video, text, audio, depth (3d), thermal (infrared radiation), and inertial measurement units (imu) readings. we find that the capabilities of pandagpt include but are not limited to: ## conclusion and limitations in this study, we present pandagpt, the first general-purpose model capable of instructionfollowing data from six modalities. the training of pandagpt is designed in a parameter-efficient way on a 160k vision-language instruction-following dataset. through extensive and concrete examples, we demonstrate that pandagpt possesses strong vision-language capabilities and such capabilities are transferable to other modalities in an emergent, i.e. zero-shot, way. despite the amazing capabilities in handling multiple modalities and their combinations. there are multiple ways to further improve pandagpt. lastly, we would like to note that pandagpt is a research prototype and cannot be readily used for real-world applications.