MAGMA – Multimodal Augmentation of Generative Models through Adapter-based Finetuning (2022)

Abstract: Large-scale pretraining is fast becoming the norm in Vision-Language (VL) modeling. However, prevailing VL approaches are limited by the requirement for labeled data and the use of complex multi-step pretraining objectives. We present MAGMA - a simple method for augmenting generative language models with additional modalities using adapter-based finetuning. Building on Frozen, we train a series of VL models that autoregressively generate text from arbitrary combinations of visual and textual input. The pretraining is entirely end-to-end using a single language modeling objective, simplifying optimization compared to previous approaches. Importantly, the language model weights remain unchanged during training, allowing for transfer of encyclopedic knowledge and in-context learning abilities from language pretraining. MAGMA outperforms Frozen on open-ended generative tasks, achieving state of the art results on the OKVQA benchmark and competitive results on a range of other popular VL benchmarks, while pretraining on 0.2 % of the number of samples used to train SimVLM.

## introduction self-supervised representation learning with transformer models @xcite has become the dominant technique in natural language processing in recent years, with encoder transformer models trained using a masked language modeling (mlm) objective @xcite excelling at natural language understanding tasks, and autoregressive decoder models @xcite @xcite displaying impressive natural language generation at increasingly large scales. vision language (vl) modeling -the modeling of joint image-text representations for tasks such as image captioning or visual question answering (vqa) -has followed suit, with the transformer encoder becoming the prevalent architecture in recent research. a popular approach among the latest state of the art vl models is to use a bert-style encoder language model (lm) in combination with an object detection backbone such as faster-rcnn @xcite . this approach, while displaying impressive performance on challenging benchmarks, has a number of drawbacks (see section 2), in particular not being able solve vl tasks in an open-ended, generative fashion. a recent line of work @xcite @xcite explores vl modeling using autoregressive decoder models trained with a language modeling objective. simvlm @xcite shows impressive performance, but requires prohibitively large amounts of pretraining data and the training of language and vision components in tandem. frozen @xcite shows that a pretrained autoregressive language model can, without any finetuning to the lm weights themselves, be harnessed to train a visual prefix which enables images to be used as its input. while the performance of frozen on vl benchmarks falls short compared to the state of the art, we feel the approach is promis-ing due to its practicality, and the public availability of large, pretrained lms such as gpt-j @xcite , pangu-α @xcite , and gpt-neo @xcite . extending the frozen approach, in this paper we introduce a framework to combine existing unimodal language and unimodal vision models pretrained on large web datasets into a powerful multimodal model. specifically, our contributions are: i) we introduce magma: an autoregressive vl model that is able to generate text from an arbitrary combination of visual and textual input. like frozen, we start from a fixed large lm and a visual encoder-prefix stack. magma differs from frozen by additionally augmenting the lm with adapter layers, and using clip's @xcite visual component as encoder. only training the adapters and visual components, the method is parameter efficient and naturally retains the lm's encyclopedic knowledge and in-context learning abilities. ii) pretrained on a simple next token prediction objective, magma is competitive in several vl downstream tasks, significantly outperforming its predecessor, frozen, while pretraining on ∼0.2 % of the number of samples used for simvlm @xcite . in particular, magma achieves state of the art accuracy on the okvqa benchmark, which we evaluate as a fully open-ended generative task. iii) our extensive ablations on the vision encoder and adapter components show i) that a pretrained clip resnet encoder outperforms other visual backbones, ii) that an adaptertuned model outperforms a visual prefix-only method, and iii) that different adapter configurations excel at different downstream tasks. iv) we show that a carefully curated pretraining dataset -including around 25 million imagetext pairs from a wide range of sources, including downstream task training data -can dramatically increase downstream performance when compared to a noisier, web-scraped dataset (cc12m @xcite ). we only explore the vl domain in this work, but we expect the general method of a modalityspecific prefix in combination with adapter layers and a frozen lm to apply equally well to other combinations of modalities, such as audio-text pairs. with this publication, we open-source our code and release a trained model checkpoint. foot_0 2 related work vl models of the past years @xcite @xcite @xcite harness a bert-like encoder transformer as the language component, trained with a mlm objective -where random words in the input are masked out, and the model is tasked with predicting them. encoder vl models are often also pretrained with auxiliary objectives or custom cross-modal losses, such as the masked region modeling, image-text matching and word-region alignment of uniter @xcite , or the contrastive loss of oscar @xcite . using auxiliary cross-modal loss functions and pretraining tasks complicates the pretraining procedure by requiring these losses to be properly balanced. additionally, encoder models need extra task-specific finetuning for each task to perform effectively, limiting their accessibility. in comparison, autoregressive vl models like magma are trained on a single, simple next token prediction objective, and can perform well on a wide range of tasks without further finetuning. two predecessors to our method are frozen @xcite and simvlm @xcite , two autoregressive decoder models trained with a next token prediction language modeling objective. frozen affixes an nfresnet @xcite vision encoder to a pretrained autoregressive lm and, keeping the lm weights frozen, trains the vision encoder along with a visual prefix that linearly maps the output of the vision encoder to the dimensionality of the lm's token embeddings. frozen shows that autoregressive vl models have the ability to adapt to examples in-context, like their language only counterparts (brown et al., 2020), without performing any gradient updates. when shown multiple examples of a task in its context window in few-shot learning, its performance on that task improves, it appears to 'learn' from the presented examples without task-specific finetuning. our model has similar in-context learning capabilities, but the addition of adapters and the different choice of visual backbone results in a model with improved performance when trained on a comparable dataset, see section 3.3. v e v p " a picture of " e v e v p " a picture of " e e n ... e 3 e 2 e 1 a beaut iful goose φ w up w down adapter block attn ff transformer block ×n vision and language components in tandem using a prefix lm objective. simvlm consists of an encoder-decoder transformer with a combined resnet @xcite and vit @xcite backbone as encoder, and a transformer decoder for language modeling. it extends the state of the art on a wide range of benchmarks, showing that a simple language modeling task can outperform mlm approaches. however, the joint pretraining requires prohibitively large uni-and multimodal datasets (1.8 billion+ image-text pairs and ∼800gb of raw text), and long training times (∼4 billion image-text pairs and ∼130 billion text tokens). aside from using orders of magnitude less data than simvlm, magma allows for the full recovery of the underlying lm's performance by simply removing the adapter layers. our work builds on recent advances in parameter efficient finetuning of lms @xcite @xcite @xcite , specifically with adapter layers @xcite @xcite , which are small modules inserted in between the elements of a transformer layer which are finetuned instead of the model weights as a form of parameter efficient fine-tuning. for a visual backbone, it is common to use region features from a pretrained object detection model such as faster- @xcite . these are generally trained using expensive human labeled data on a bounded set of object classes, limiting the number of object types the resulting model can recognize. on the other hand, contrastive models such as clip @xcite and align @xcite present a more robust approach to learning visual features by learning joint representations between image-text pairs. they show strong performance on a wide variety of vision tasks as well as impressive generalization abilities that can provide powerful semantic guidance to image generation @xcite . but since they were only trained to match image-text pairs, they cannot inherently be used for tasks that require text generation as output @xcite . however, @xcite show that the weights of contrastive language-image models contain useful semantic information for vl tasks. by replacing the conventional region-based backbone with clip's visual encoder in popular vl architectures, the authors achieve sota results across a wide variety of vl tasks without needing regionbased features, motivating us to use clip's visual component as a vision encoder for magma. notably, we confirm their finding that the vit variant of clip underperforms on vl tasks when compared to the resnet variant, particularly in tasks that require localization within an image. ## method our general approach is an image conditioned variant of soft-prompting or prefix tuning @xcite @xcite for language transformers and extends the frozen method @xcite . the core idea is to translate image features into language embeddings carrying visual information which can therefore be interpreted by the language transformer without need to retrain the latter from scratch. ## experiments and analysis to evaluate our methodology, we first train a series of ablations (cf. §4.2), to break down the effects of the vision encoder and adapter choice. we evaluate these ablations, and all subsequent models on a range of visual question answering and image captioning tasks designed to quantify the model's ability to adapt to new tasks using in-context learning, recognize a wide variety of objects, and reason in detail about an image -often involving complex spatial understanding, encyclopedic world knowledge, and optical character recognition (ocr). ## conclusion we propose a simple framework for multimodal augmentation of generative models through adapter-based finetuning -demonstrating that it is possible to transform multiple unimodal models into a powerful multimodal vl model while keeping the weights of the language component frozen. our model, magma, trained using adapter layers and a simple next token prediction objective, can perform competitively with state of the art vl models on a wide range of benchmarks, excelling at tasks requiring external knowledge and the recognition of uncommon object classes. we hope our results will act as a starting point for further research into augmenting pretrained language models with additional modalities.