A ny MAL : An Efficient and Scalable Any-Modality Augmented Language Model (2024)

Abstract: We present Any-Modality Augmented Language Model (AnyMAL), a unified model that reasons over diverse input modality signals (i.e. text, image, video, audio, IMU motion sensor), and generates textual responses. AnyMAL inherits the powerful text-based reasoning abilities of the state-of-the-art LLMs including Llama-3 (70B), and converts modality-specific signals to the joint textual space through a pre-trained aligner module.In this paper, we provide details on the optimizations implemented to efficiently scale the training pipeline, and present a comprehensive recipe for model and training configurations. We conduct comprehensive empirical analysis comprising both human and automatic evaluations, and demonstrate state-of-the-art performance on various multimodal tasks compared to industry-leading models – albeit with a relatively small number of trainable parameters.

## introduction large language models (llms), known for their substantial size and complexity, have significantly enhanced the capacity of machines to understand and articulate human language. the progress in llms has also led to notable advancements in the vision-language domain @xcite @xcite , bridging the gap between image encoders and llms to combine their reasoning capabilities. prior multimodal llm research has concentrated on models that combine text and one other modality @xcite , such as text and image models, or has centered on proprietary language models that are not open sourced @xcite . to tackle the previously mentioned challenges, we introduce any-modality augmented lan-guage model (anymal) -a collection of multimodal encoders trained to transform data from various modalities, including images, videos, audio, and imu motion sensor data, into the text embedding space of an llm. to achieve this, we extend the work by @xcite to ( 1 ) more capable instruction-tuned llms (i.e. llama-3-70b-chat (ai@meta, 2024)), (2) larger pre-trained modality encoders, and (3) advanced projection layers to handle variable input lengths. the output examples are shown in figure 1 , and an illustration of the methodology is in figure 2 . the key contributions of the work are as follows: ## related work large language models (llm): there has been a surge of llms with varying model sizes recently, showcasing remarkable reasoning capabilities. while the most well-known commercial service is gpt4 (openai, 2023), the publicly released models include @xcite , opt @xcite , llama 1 & 2 @xcite , vicuna @xcite , and more recently, llama-3 (ai@meta, 2024). our work builds upon the powerful text-based reasoning capabilities of these llms, extending these capabilities to multimodal inputs. vision-language models: numerous studies have addressed the task of instructing a unified model that integrates both visual and linguistic elements, finding practical implementations in domains like image captioning @xcite and visual question answering (vqa) tasks @xcite @xcite . while the relative scarcity of data sources aligning different modalities has conventionally been considered the bottleneck in scaling, recent works have shifted towards harnessing the capabilities of pretrained llms, tapping into the knowledge accrued from extensive textual corpora. these work include flamingo @xcite , openflamingo @xcite , palm-e @xcite , blip-2 @xcite , instructblip @xcite , llava @xcite , idefics (laurençon et al., 2023), minigpt-4 @xcite and many more @xcite @xcite @xcite @xcite , where each model uses different variants of base llms. these models typically undergo fine-tuning stages as well, re-purposing several task-specific vision-language datasets @xcite . our work extends the previous approaches by (1) allowing for diverse input modalities beyond vision signals, (2) presenting a fine-tuning process with our manually collected multimodal instruction tuning and human preference data, and (3) scaling the llm parameters to 70b via an efficient pretraining approach. ## experiments given the high-level of alignment among the modalities, we evaluate the model's reasoning and instruction-following abilities which it inherits from the core instruction-tuned llm, as well as from the multimodal instruction-tuning process. we conduct a comprehensive comparison with strong baseline models for each respective modality pair (vision-language and audio-language) from the open-sourced literature and industry. vqa benchmarks: overall, our anymal exhibits competitive performance compared to the industry-leading models (e.g. gemini, gpt4) across multiple tasks, despite the relatively small number of parameters. among the base llm models for anymal, 70b shows the most robust performance, underscoring the influence of substantial reasoning proficiency inherent in larger llms on tasks involving visual reasoning. human evaluation on image-based reasoning tasks: we evaluate the performance of our models against the most competitive vision-language models publicly available to run inference on (i.e. gemini 1.5 pro @xcite , and claude 3 ## conclusions our proposed anymal showcases a novel and natural way of interacting with an ai model, e.g. asking questions that presume a shared understanding of the world between the user and the agent, through the same lens and combinatory perceptions (e.g. visual, auditory, and motion cues). the proposed scalable way of training anymal makes it possible to leverage the powerful reasoning capabilities of llms within the multimodal settings.