Frontier Review of Multimodal AI (2023)

Abstract: “Pre-training techniques have enabled foundation models (such as BERT, T5, GPT) to achieveremarkable success in natural language processing (NLP) and multimodal tasks that involve text,audio and visual contents. Some of the latest multimodal generative models, such as DALL·Eand Stable Diffusion, can synthesize novel visual content from text or video inputs, which greatlyenhances the creativity and productivity of content creators. However, multimodal AI also facessome challenges, such as adding new modalities or handling diverse tasks that require signalsbeyond their understanding. Therefore, a new trend in multimodal AI is to build a compositionalAI system that connects existing foundation models with external modules and tools. This way,the system can perform more varied tasks by leveraging different modalities and signals.Inthis paper, we will give a brief overview of the state-of-the-art multimodal AI techniques and thedirection of building compositional AI systems. We will also discuss the potential future researchtopics in multimodal AI.”

## introduction large language models (llms) have achieved great success in natural language processing (nlp). these models (e.g., bert @xcite , t5 @xcite and @xcite ) can learn general data representations and commonsense knowledge from large-scale corpora using self-supervised learning tasks (such as masked language modeling or next token prediction). the learned models can be further fine-tuned on downstream tasks and obtain superior performance on them. the success of llms has also been extended to other non-language domains, such as computer vision or speech processing. the convergence of these techniques on different types of data makes "multimodal ai" the hottest direction in the ai community. this paper aims to briefly summarize the latest trends of multimodal ai research. in short, there are three trends as follows: (1) the underlying architectures of models for different modalities are converging; ## the convergence of model architecture the architecture of models for different modalities is becoming more similar in the era of llms. transformers are widely used in text, code, visual and audio scenarios to support understanding and generation tasks. for instance, the latest llms like chatgpt or gpt-4 have integrated text and code in a single model, which can support text-only, code-only, text-to-code and code-to-text generation tasks. multimodal generation models like dall•e @xcite and nuwa-infinity @xcite are also trained based on auto-regressive models like gpt models for image and video generation tasks. vall-e (wang et al., 2023) can leverage strong in-context learning capabilities and can be applied for zero-shot cross-lingual text-to-speech synthesis and zero-shot speech-to-speech translations, which is also based on transformer and gpt-like models. moreover, we also observed that diffusion models are widely used in content generation tasks as well, such as dall•e 2 (ramesh et al., 2022) and stable diffusion (rombach et al., 2022) for visual generation, naturalspeech 2 (shen et al., 2023) for speech generation. but there is also another research thread that aims to unify different types of generation models using diffusion models, which can be also seen as an indication of the model architecture convergence. due to the different basic units, data formats, and structures of the contents in different modalities, there is still no universally agreed model architecture for multimodal ais. however, such convergence is definitely a clear trend in the ai community. ## from visual-language understanding to visual generation from language visual-text (vl) pre-trained models are the most representative multimodal ais. the goal of such models is to learn the representations of texts and visuals jointly and support vl tasks such as image retrieval, visual question answering, or text-based image generation. in the past several years, the research focus has shifted from vl understanding tasks to visual generation tasks. therefore, in this section, we will first review the progress of vl understanding models and then review the latest development of visual generation models from texts. ## future directions this paper briefly reviews the recent developments of multimodal ai research, including (1) the model architectures are becoming more similar, (2) the research focus is moving from multimodal understanding models to multimodal generation models; (3) combining llms with external tools and models to accomplish diverse tasks is emerging as the new ai paradigm. there are several directions that can be further explored in the future. first, concentrating more on video generation, which could trigger the next chatgpt breakthrough in the ai community. second, concentrating more on compositional ai for multimodal systems with more modalities covered and less computation costs needed. third, concentrating more on the autonomous robotics, which can complete more tasks in the physical world, to further enhance human's creativity and productivity.