i-Code Studio: A Configurable and Composable Framework for Integrative AI (2024)

Abstract: Artificial General Intelligence (AGI) requires comprehensive understanding and generation capabilities for a variety of tasks spanning different modalities and functionalities. Integrative AI is one important direction to approach AGI, through combining multiple models to tackle complex multimodal tasks. However, there is a lack of a flexible and composable platform to facilitate efficient and effective model composition and coordination. In this paper, we propose the i-Code Studio, a configurable and composable framework for Integrative AI. The i-Code Studio orchestrates multiple pre-trained models in a finetuning-free fashion to conduct complex multimodal tasks. Instead of simple model composition, the i-Code Studio provides an integrative, flexible, and composable setting for developers to quickly and easily compose cutting-edge services and technologies tailored to their specific requirements. The i-Code Studio achieves impressive results on a variety of zero-shot multimodal tasks, such as video-to-text retrieval, speech-to-speech translation, and visual question answering. We also demonstrate how to quickly build a multimodal agent based on the i-Code Studio that can communicate and personalize for users. The project page with demonstrations and code is at https://i-code-studio.github.io/.

## introduction large language models (llms) such as bert @xcite and gpt-3 @xcite , visual-language models (vlms) like clip @xcite and dall-e @xcite , and audio language models (alms) such as w2v-bert @xcite have enabled a variety of capabilities, from zero-shot image classification to reading comprehension, automatic speech recognition, and photorealistic image generation. the performance and capability of these pre-trained models are, however, influenced by the data they are exposed to, which varies across different domains; llms are trained on diverse sources of data, such as webpages, novels, and wikipedia corpora, while vlms are trained on pairs of images or videos and their captions, and alms are trained on audio data such as speech. these distinct training domains render the pre-trained models different and sometimes complementary capabilities. for instance, llms are suitable for tasks such as reading comprehension but unable to interpret audio and visual information; vlms can produce photorealistic images but cannot tackle complex language understanding. on the other hand, humans can often easily handle distinct tasks like the above with multimodal input and output. therefore, in order to build artificial general intelligence (agi), we need to break the barriers between modalities and specific tasks. instead of building a single model to handle all possible tasks, which is infeasible under current technology, a lot of research has recently emerged to focus on the composition of large pre-trained models to achieve integrative ai, either via finetuning them jointly on new tasks @xcite @xcite , or via a shared modality, such as language, to capture new multimodal capabilities without the need for finetuning @xcite @xcite . issues with these approaches are 1) there often lacks data and computation resources for joint finetuning, and 2) one cannot easily configure and compose different large pre-trained models in an agile framework to adapt to different needs. therefore, in this paper, we propose i-code studio, a configurable and composable framework for integrative ai (figure 1 ) . the i-code studio allows developers to quickly and easily orchestrate various cutting-edge pre-trained models in a finetuning-free fashion. these pre-trained models are from different modalities, and the strength of each individual model is integrated to conduct complex multimodal tasks. for each task, a directed acyclic graph (dag) is configured so that the related models cooperate to produce the desired output. the input data flows through each node in the dag, enabling complex multimodal tasks to be completed. this makes i-code studio an integrative, flexible, and composable framework. for instance, for visual question answering task, a dag is configured using the input image, the input question, the florence @xcite vision foundation model, a language prompt, the chatgpt, and an output, each represented by a node. the visual information from the input image is fed into florence. the florence node processes the image and outputs a set of detected object categories/tags and a caption. these outputs and the input question are then fed into a node that generates a vlm-informed language prompt. finally, this cross-modal prompt is used by chatgpt to generate an answer to the input question which is sent to the output node. in this paper, we showcase the effectiveness of the i-code studio using models from azure cognitive services (acs) and openai services. the resulting integrative model achieves the state-ofthe-art (sota) or comparable to the sota performance on zero-shot tasks such as speech-to-speech translation, video-to-text retrieval, and visual question answering. we also show how to quickly build a multimodal agent to interact with a user. in summary, our main contributions are the following: ## related work recently, the composition of large pre-trained models has been extensively studied. the most common way to compose these models is to fine-tune them jointly on new tasks. @xcite proposed unit, a unified transformer model that is capable of learning several tasks across multiple domains, including object detection and multimodal reasoning. this model is based on a transformer encoder-decoder architecture, where each input modality is encoded with an encoder, and shared decoders are used to make predictions for each task. @xcite proposed a vision-language pretraining framework, called simvlm that is trained end-to-end with a single language modeling objective. the simvlm reduces the complexity of training by utilizing weak supervision on a large scale. @xcite proposed flamingo, a collection of vlms that can connect pre-trained vision-only and language-only models, process sequences of interleaved visual and textual data, and accept images or videos as inputs. however, these methods can be computationally expensive. the i-code studio differs from these approaches since it does not require finetuning, which enables the fast composition of pre-trained models for a variety of tasks and reduces the time and expense associated with finetuning. unlike these work, models can be composed via a shared modality, such as language. @xcite combined a visual-semantic model with a large language model, enabling the models to take advantage of the knowledge present in both webscale models for image caption generation task. more related to our work, @xcite proposed socratic models, a modular framework that enables multiple pre-trained models to exchange information with each other, capture new multi-modal capabilities without the need for finetuning, and be composed without any prior training using multimodal-informed prompting. our i-code studio is a more integrative, flexible, and composable framework compared to these work, allowing users to compose cutting-edge models and technologies customized for their particular needs easily. distinct from the work mentioned, @xcite proposed a closed-loop approach to combining pre-trained models in such a way that they act as generators and scorers. the generators create proposals, while the scorers provide feedback to improve the generated results. this type of iterative consensus optimization allows models to correct mistakes made by other models, leading to significant improvements in downstream tasks. @xcite studied the application of llms in embodied environments for robotic control. they combined llms with different sources of text feedback and found that natural language acts as a universal means of communicating with the model. the resulting system, called inner monologue, integrates various components such as perception models, robotic skills, and human feedback to effectively execute user commands. ## the i-code studio framework in this section, we introduce i-code studio, a configurable and composable framework for integrative ai. given a complex multimodal task, the i-code studio provides a generic framework for developers to quickly and easily integrate and compose several large pre-trained models and services across different modalities without any training or finetuning to accomplish the task. figure 2 shows examples of building ai solutions for various multimodal tasks using the i-code studio framework. for each task, the framework can be represented via a dag, where the nodes with no incoming edge are the raw input data such as image, text, video and speech, the nodes with no outgoing edges are the outputs of the given task, and the rest of the nodes are foundation models/services or hold intermediate model outputs from other models/services. the input to a node comes from the raw input, and/or the output from previous nodes. the input data flows through each node in the dag, enabling complex multimodal tasks to be completed. an outgoing edge from a model/service node represent an api provided by the model/service. for each task, the inputs enter the dag from the input nodes, figure 2 : the i-code studio can be used to build ai solutions for various multimodal tasks. for each task, a dag is configured so that the related models cooperate to produce the desired output. the input data flows through each node in the dag, enabling complex multimodal tasks to be completed. the input nodes are represented by double blue circles, the model/service nodes, e.g. chatgpt and florence, by black circles, the output nodes by double black circles, and the rest by dash-dotted red circles. see the text for details about each multimodal task. and are processed by one or more models or model services. in the process, edges convert the format of a module's output, filters data, or apply an api such as summarization, translation, object detection, image captioning, transcribing, text-to-speech synthesis, etc. for each task, a dag is configured so that the related models cooperate to produce the desired output. the different components of i-code studio cooperate seamlessly to form a single, integrated ai solution, and can be adjusted to fit the specific needs of the user. for instance, for visual questions answering (vqa) task the input is an image and a question related to the image (see figure 2 ). we can first apply image captioning and object detection services to the input image. the output text, which contains the visual information, is merged with the input question as the prompt to chatgpt, which answers the question. for speech-to-speech translation, the dag is configured with speech recognition (sr) → machine translation (mt) → text-to-speech (tts). this dag transcribes the source speech, translates the transcription into the target language, and generates the target speech. to build i-code studio, we utilize azure machine learning studio, a cloud-based, collaborative, drag-and-drop development environment for building, testing, and deploying machine learning models. we encapsulate available models and services from azure cognitive services (acs) as independent apis and deploy them as an integrated web service for real-time invoking. in this way, it allows developers to flexibly combine them to build their own applications. more details about the available foundation models and services are presented in appendices a and b. ## evaluations in this section, we presents our experiments in three tasks covering language, speech and vision modality: 1) video-to-text retrieval; 2) visual question answering and 3) speech-to-speech translation. ## applications: multimodal agents as humans, we have a complex sensory system that allows us to experience the world around us. we use our eyes to see, ears to hear, mouths to talk, and brains to process and interpret the information we receive. inspired by this, we utilize i-code studio to build a multimodal agent that can communicate and personalize for users. specifically, the eyes of the agent use azure vision services to interpret visual images signals and send signals to the brain; the ears and mouth use azure speech services to collect sound waves and produce sounds; the brain leverage azure openai services to integrate all the sensory signals received from the eyes, ears and uses them to make decisions. this interconnected system of sensory organs and the brain are what enables our multimodal agents to understand and interact with the world around us. our multimodal agent is a virtual assistant with "eyes" (florence), "ears" (acs asr), "brain" (gpt-3) and mouth (acs tts). the i-code studio integrates speech and vision signals from users by composing and configuring services from acs and openai. figure 5 shows a demo example. using language prompting, i-code studio can enable multimodal dialogue between the user and agent. guis call i-code studio once to simplify the developing cost while giving consistent user experience. ## conclusion the i-code studio, is a new configurable and composable framework for integrative ai. it orches-trates multiple pre-trained models to conduct complex multimodal tasks, without the need for finetuning. we showed the i-code studio can achieve impressive results on three multimodal tasks. we also demonstrated how to build a multimodal virtual assistant agent with the i-code studio. with further research and development, the i-code studio can be extended to be more flexible and powerful to create even more complex applications.