A ny GPT : Unified Multimodal LLM with Discrete Sequence Modeling (2024)

Abstract: We introduce AnyGPT, an any-to-any multimodal language model that utilizes discrete representations for the unified processing of various modalities, including speech, text, images, and music. AnyGPT can be trained stably without any alterations to the current large language model (LLM) architecture or training paradigms. Instead, it relies exclusively on data-level preprocessing, facilitating the seamless integration of new modalities into LLMs, akin to the incorporation of new languages.We build a multimodal text-centric dataset for multimodal alignment pre-training. Utilizing generative models, we synthesize the first large-scale any-to-any multimodal instruction dataset. It consists of 108k samples of multi-turn conversations that intricately interweave various modalities, thus equipping the model to handle arbitrary combinations of multimodal inputs and outputs.Experimental results demonstrate that AnyGPT is capable of facilitating any-to-any multimodal conversation while achieving performance comparable to specialized models across all modalities, proving that discrete representations can effectively and conveniently unify multiple modalities within a language model. Demos are shown in https://junzhan2000.github.io/AnyGPT.github.io/.

## introduction llms @xcite @xcite have exhibited remarkable proficiency in comprehending and generating human language. nevertheless, their capabilities are confined to textual processing. the real-world environment is inherently multimodal, with organisms perceiving and exchanging information through diverse channels, including vision, language, sound, and touch. a promising objective in developing multimodal systems is to augment llms with the capacity for multimodal perception @xcite . the dominant methodology involves the integration of multimodal encoders with the language model, thus empowering it to process information across various modalities and utilize its sophisticated text-processing abilities to produce coherent responses. however, this strategy is limited to text generation and does not encompass multimodal output. pioneering efforts such as emu @xcite , seed-llama @xcite and speechgpt @xcite have made significant strides by enabling multimodal understanding and generation within language models. yet, these models incorporate only a single non-textual modality, such as images or audio. while aligning text with one additional modality is relatively straightforward, integrating multiple modalities (@xmath0. existing explorations in any-to-any multimodal generation have encountered obstacles: some @xcite lacked a robust core language model, which impeded the system's reasoning and decision-making capabilities; others, such as next-gpt @xcite , codi-2 @xcite , and unified-io2 @xcite , have employed separately pre-trained encoders and decoders. this approach results in representational inconsistencies between the inputs and outputs of the llms, which in turn complicates both training and inference processes. moreover, stabilizing training with such diverse modalities necessitates substantial modifications to existing models and techniques. to overcome these challenges, we introduce anygpt, an any-to-any multimodal language model that employs discrete representations for unified processing. anygpt is equipped with multimodal tokenizers that compress raw multimodal data, such as images and audio, into a sequence of discrete semantic tokens. these discrete representations enable the core llm to unify tasks such as perception, understanding, reasoning, and generation in an autoregressive manner at the semantic level. subsequently, de-tokenizers convert the discrete representations back into the original modal representations at the perceptual level. thanks to discrete representation, which filters out highfrequency, modality-specific perceptual information while preserving essential low-frequency semantic information @xcite @xcite , we can train our model stably without any alterations to the existing llm architecture or training paradigms. instead, our approach relies solely on data-level preprocessing. this allows for the seamless integration of new modalities into llms, akin to the addition of new languages, and permits the direct application of existing llm tools, which enhances the efficiency of both the training and inference stages. furthermore, to mitigate the scarcity of multimodal alignment data encompassing all modalities, we build a text-centric multimodal alignment dataset for pre-training. our goal is to use text as a bridge, by aligning other modalities with text, to achieve mutual alignment among all modali-ties, since natural language is the most refined modality of semantic representation and is present in the majority of multimodal alignment datasets. to endow the model with the capability to comprehend and generate content interwoven with multiple modalities, we employ advanced generative models to synthesize a multimodal instruction dataset, anyinstruct-108k. this dataset, comprising 108k samples of multi-turn conversations, enables anygpt to handle arbitrary combinations of multimodal inputs and outputs. experimental results demonstrate that anygpt achieves zero-shot performance comparable to that of specialized models across various modalities. furthermore, extensive case studies corroborate anygpt's remarkable ability to facilitate any-toany multimodal dialogue, substantiating the feasibility of using discrete representations to unify multiple modalities. our contributions include the following: ## anygpt our interest lies in facilitating the generation of any modality to any modality with llms. to realize this, we propose a comprehensive framework that can be uniformly trained. as illustrated in figure 1 , this framework is composed of three main components: (1) multimodal tokenizers, (2) a multimodal language model serving as the backbone, and (3) multimodal de-tokenizers. the tokenizers transform continuous non-text modalities into discrete tokens, which are subsequently arranged into a multimodal interleaved sequence. then the sequences are trained by the language model using the next token prediction training objective. during the inference process, multimodal tokens are decoded back into their original representations by the associated de-tokenizers. to enrich the quality of generation, multimodal enhancement modules can be deployed to post-process the generated results, including applications like voice cloning or image super-resolution. in the following section, we will introduce the details of each module. image tokenizer we utilize the seed tokenizer @xcite ) for image tokenization. ## conclusion in this work, we introduced anygpt, an any-toany multimodal language model that utilizes discrete representations for the unified processing of various modalities, including speech, text, images, and music. discrete multimodal representations facilitate a seamless integration of new modalities-comparable to incorporating a foreign language-without necessitating alterations to the existing llm architecture or training paradigms. to equip the model to handle arbitrary combinations of multimodal inputs and outputs, we synthesize the first large-scale any-to-any multimodal instruction dataset, anyinstruct-108k, consisting of multi-turn conversations that intricately interweave various modalities. experimental results indicate that anygpt achieves promising results in various cross-modal tasks and demonstrates that discrete representations can effectively and conveniently unify multiple modalities within a unified large language model.