i-Code V2: An Autoregressive Generation Framework over Vision, Language, and Speech Data (2024)

Abstract: The convergence of text, visual, and audio data is crucial towards human-like artificial intelligence, however the current Vision-Language-Speech landscape is dominated by encoder-only models that lack generative abilities. We propose closing this gap with i-Code V2, one of the first models capable of generating natural language from any combination of Vision, Language, and Speech data. i-Code V2 leverages state-of-the-art single-modality encoders, combining their outputs with a new modality-fusing encoder to project combinations of modalities into a shared representational space. Language tokens are generated from these representations via an autoregressive decoder. i-Code V2 is pretrained end-to-end on a large collection of dual- and single-modality datasets with a novel text completion objective that can be generalized across arbitrary combinations of modalities. i-Code V2 matches or outperforms state-of-the-art single- and dual-modality baselines on 7 multimodal tasks, demonstrating the power of generative multimodal pretraining across a diversity of tasks and signals.

## introduction pretrained large language models (llms) have experienced massive success as general-purpose solutions for multiple tasks @xcite . however, a large gap persists between the capabilities of llms and true humanlike intelligence. this is partially because humans perceive a variety of sensory inputs while llms are typically restricted to language (l) data and unable to understand or generalize to other modalities such as vision (v) and speech audio (s). recently, the field of multimodal ai, which aims to develop ai systems capable of modeling multiple kinds of signals, has witnessed significant progress including new learning techniques @xcite @xcite , training data @xcite @xcite , and model architectures @xcite @xcite . despite this progress in multimodal ai, most research has focused on understanding pairs of modalities, such as speech-language and visionlanguage, and the fast-growing subfield of triplemodality ai (language, vision, speech) remains limited to encoder-only models @xcite @xcite . this paper proposes i-code v2, one of the the first encoderdecoder generative models for the triple-modality setting. i-code v2 can flexibly generate text from arbitrary combinations of language, vision, and speech data. this model addresses three ongoing challenges within multimodal research. first, most existing vision-language-speech models are encoder-only, i.e. they can conduct discriminative tasks such as multimodal classification but not generative ones like visual question answering or automatic speech recognition. i-code v2 enables the model to generate content from multimodal signals, unlocking more diverse applications and improved discriminative performance. second, most existing triple-modality research leverages triple-modality data (i.e. video with subtitles and audio track). however, the three modalities in video data can be noisily aligned @xcite which degrades downstream pretraining. furthermore, the available high quality video data is several orders of magnitudes smaller in size than single-or dual-modality ones. e.g., the largest publicly available image-caption dataset laion @xcite has 5 billion pairs (335 billion text tokens) while the largest video dataset merlot has 180m videos (5 billion text tokens) @xcite . i-code v2 proposes a novel method for efficiently leveraging these larger and higher-quality dual-and single-modality datasets within a triple-modality pretraining framework. we accomplish this with a new, generalized sequence-to-sequence pretraining objective which unifies assorted multimodal objectives into simple text completion. third, multimodal tasks are diverse in settings and data formats, e.g. automatic speech recognition (asr), vision qa, sentiment analysis, etc. existing techniques apply separate inference strategies to each problem type, adding complexity and overhead for practitioners. i-code v2 unifies all tasks under its text completion framework, rendering multimodal inference and cross-task transfer easier for practitioners. we evaluate i-code v2 on 7 datasets: multimodal summarization, multimodal dialogue generation, multimodal sentiment analysis, vision qa, vision captioning, and asr. notably, i-code v2 outperforms previous sota models on msmo (multimodal summarization), image chat (multimodal dialogue generation), ur-funny (multimodal sentiment analysis). i-code v2 also exhibits competitive performance compared to specialized dual-modality models on vision qa, vision captioning, and asr, suggesting the power of integrative multimodal pretraining. in summary, our key contributions are threefold: ## related work multimodal learning studies extracting and incorporating information from vision, language, and speech modalities. a recent advance is unifying models of different modalities to the transformer. for example, representing vision and language with one multimodal transformer model has shown great performance in image caption @xcite , vision classification @xcite , vision question answering @xcite , etc. extracted image features @xcite or projections of image patches @xcite are fused together with text token embeddings, then input to a multimodal encoder to obtain unified representations for vision and language. for vision-language-speech models, the multimodal encoder is pretrained on video data @xcite or dual-modality data pairs @xcite . multimodal representations can be integrated by a late-stage multimodal fusion network @xcite , or integrated early at the input stage @xcite . generative multimodal model can generate one modality from another modality or a combination of input modalities. e.g., image captioning @xcite , automatic speech recognition @xcite ), text-to-image generation @xcite @xcite , etc. several recent works propose to unify vision-language tasks with one homogeneous model architectures and schemes. for example, wang et al. (2022c); lu et al. (2022); tang et al. (2022) unite vision and vision-language tasks, such as image classification, object detection, semantic segmentation, visual qa, document understanding, image generation, etc. huang et al. (2023) recently proposed the kosmos-1 model to generate text based on vision and text input. distinct from previous works, i-code v2 can not only encode and merge vision, language, and speech modalities, but also generate natural language. it unifies various tasks across multimodal summarization, multimodal sentiment analysis, speech recognition, visual qa, and caption. ## experiments we test i-code v2 on 7 datasets from assorted categories. in downstream tasks, we update parameters of vision-language-speech encoder, language decoder, and single-modality encoders. overall, i-code v2 sets a new state-of-the-art in 3 tasks (msmo, image chat, and ur-funny) and remains highly competitive in the rest, suggesting the promise of integrative and generative multimodal pretraining. ## analysis & explorations ablation study on pretraining effectiveness. we investigate the pretraining effectiveness by comparing performance of i-code v2 with and without the multimodal pretraining (section 3.2). as shown in pretraining objectives ablation. we explore how pretraining objectives affect model's performance. as shown in zero-shot learning. we test pretrained i-code v2 on vizwiz-vqa without finetuning. as an open-vocabulary generative model, the zero-shot performance of i-code v2 is respectful, with overall accuracy 22.53%, and 73.6% for "yes/no" answers, 8.47% for "number" answers, 24.46% for "other" answers, 10.49% for "unanswerable" answers respectively (for reference, flamingo-9b zero-shot accuracy is 28.8%). note that vizwiz-vqa questions are from visually impaired population and images are also distinct from those in vqa data used in pretraining. this performance indicates that i-code v2 can closely follow the task instruction to answer the question. it also shows that i-code v2 learns to answer visually grounded questions from pretraining, even though there are assorted pretraining tasks and datasets. training hyperparameters. in ## conclusion in this paper, we propose i-code v2, a multimodal generative model that jointly encodes language,