Expedited Training of Visual Conditioned Language Generation via Redundancy Reduction (2024)

Abstract: We introduce \text{EVL}_{\text{Gen}} , a streamlined framework designed for the pre-training of visually conditioned language generation models with high computational demands, utilizing frozen pre-trained large language models (LLMs). The conventional approach in vision-language pre-training (VLP) typically involves a two-stage optimization process: an initial resource-intensive phase dedicated to general-purpose vision-language representation learning, focused on extracting and consolidating relevant visual features. This is followed by a subsequent phase that emphasizes end-to-end alignment between visual and linguistic modalities. Our novel one-stage, single-loss framework bypasses the computationally demanding first training stage by gradually merging similar visual tokens during training, while avoiding model collapse caused by single-stage training of BLIP-2 type models. The gradual merging process effectively condenses visual information while preserving semantic richness, resulting in rapid convergence without compromising performance. Our experimental findings demonstrate that our approach accelerates the training of vision-language models by a factor of 5 without a noticeable impact on overall performance. Furthermore, we illustrate that our models significantly narrow the performance gap to current vision-language models using only 1/10 of the data. Finally, we showcase how our image-text models can seamlessly adapt to video-conditioned language generation tasks through novel soft attentive temporal token contextualizing modules. Code: https://github.com/yiren-jian/EVLGen

## introduction the landscape of vision-language modeling has undergone significant transformations in recent years, with clip @xcite serving as a landmark development. it distinguished itself through unparalleled zero-shot classification capabilities and efficiency in image-text retrieval tasks from prior works @xcite . successive models like albef @xcite , x-vlm @xcite , and vlmo @xcite further broadened the scope, addressing a myriad of tasks such as retrieval, visual entailment, and closed-set visual question answering (vqa), among others. recently, the field has been enriched by the advent of generative models designed for complex image-to-language tasks. notable contributions include coca @xcite , simvlm @xcite , frozen @xcite , and flamingo @xcite , targeting tasks like image and video captioning and open-set vqa. these models all rely on billion-scale datasets for training from scratch to bridge the substantial modality gap between vision and language. as a result, the resource-intensive requirements (i.e., thousands of tpus) of these training-fromscratch vision-language models (vlms) led to the conceptualization of blip-2 @xcite : this model alleviates computational costs (e.g., only requiring 16× fewer gpus) by integrating existing well-pretrained vision encoders (vit) with language decoders (llm), and then tuning their joint operation. a central innovation in aligning vision and language modules in blip-2 is q-former, a multimodal connector equipped with learnable queries for enhancing cross-attention mechanisms. this architectural choice, however, prevents the full model from end-to-end training and therefore still demands an additional pre-training regimen for qformer, referred to as blip-2's stage 1. the stage involves three learning objectives-image-text contrastive, image-text matching, and language generation-and necessitates multiple forward passes for facilitating the q-former's optimization. despite its efficiency gains over coca, blip-2's training still imposes considerable computational costs. this poses challenges for research environments with limited computational resources, such as university labs. our experiments indicate that the stage-1 training of blip-2 took approximately eight days on eight a100-80g gpus (see appendix b for training configurations). this computational burden has consequently restricted research to using the pre-trained q-former, hindering the exploration of alternative vits in vlms. this limitation is evident in subsequent works such as instructblip @xcite , videochat @xcite , video-llama @xcite , x-llm @xcite . the prospect of reducing blip-2's computational cost through end-to-end, single-stage training is compelling. such an approach would remove the complexities associated with resource allocation and hyper-parameter tuning inherent in multi-stage training. yet, direct end-to-end training with blip-2 poses substantial challenges, corroborated by both original findings from blip-2 and our own empirical analyses. we hypothesize that these challenges emanate from the intrinsic design of the q-former. specifically, the inclusion of randomly initialized learnable queries and crossattention mechanisms complicates the optimization landscape, especially when the aim is to minimize the representational disparity between visual and linguistic modalities. in this paper, we propose a token merging transformer (tomeformer) as an efficient visionlanguage connector. tomeformer employs a systematic token-merging @xcite strategy that is both intuitive and effective. by connecting a pre-trained vit as the visual encoder and a frozen llm as the language decoder, we intro-duce a new vlm "expedited visual language generation model" (evl gen ), facilitates a streamlined, single-stage training process. it requires only a singular learning objective and a single forward pass per optimization step. this stands in contrast to blip-2's multi-stage training, laden with multiple objectives and several forward passes. further, we introduce a soft attentive temporal contextualization mechanism within the vit for effective video-language modeling. this uncovers more shared semantic features across temporal frames, thereby improving the efficiency of the spatial token merging process. it eliminates the need for modality realignment, contrasting approaches such as the temporal q-former @xcite , or the addition of new learnable temporal queries @xcite . our strategy simplifies the optimization challenges tied to working with relatively smaller video-text datasets, compared to their image-text counterparts. remarkably, we demonstrate that even without video pre-training, our temporal token contextualize approach can effectively train robust video-language models. this differs from recent work in videolanguage models @xcite ) that depend on pre-training models using vast million-scale video-text datasets. in summary, our contributions are: ## related work image-language models coca @xcite , trained on billions of image-text pairs, represents a state-of-the-art approach in generative tasks like open vqa and visual captioning. to mitigate the computational demands of pre-training, blip-2 @xcite employs frozen pre-trained vit and llm components, focusing on training a specialized connector between visual and linguistic modalities called the q-former. due to the computationally intensive nature of training blip-2, subsequent models in visual instruction @xcite @xcite have predominantly utilized the pre-trained q-former, which is aligned with the eva-vit-g model supplied by blip-2. additional related works on image-language modeling are further discussed in appendix a. video-language models while many imagetext models can be adapted for video-text tasks through simple feature pooling (e.g., @xcite ), the field has seen specialized models that incorporate temporal dynamics. building on the foundation of blip-2, video-llama @xcite enhances its architecture by introducing additional temporal q-former layers between the spatial q-former and the llm components of blip-2. inspired by blip-2, most recent works such as videochat @xcite , pandagpt @xcite , valley @xcite , and video-chatgpt (muhammad maaz and khan, 2023) leverage frozen llms in their videolanguage models. token merging token merging (tome) @xcite aims to improve the inference speed of pre-trained vits without requiring re-training. at each transformer layer, tokens are divided into two sets and subsequently merged based on similarity, effectively reducing the token count and thereby accelerating inference. this method maintains classification and generation quality. in our work, we repurpose tome to condense the visual features used as language prompts in the llm. we integrate a standard transformer with tome capabilities, resulting in a model we term tomeformer. this model serves as an effective connector between visual and language domains, preserving semantic richness while reducing token count. importantly, this integration of tome does not introduce any additional parameters. inspired by spatial tome, we introduce a novel soft temporal tome variant within the vision encoder, thereby adding temporal modeling capabilities to our image-text models. additional related works on token redundancy are further discussed in appendix a. ## methods we begin by presenting our image-text model and then describe the adaptations made to this pretrained model for video-related tasks. ## experiments our experimental setup is as follows: weight decay is set at 0.05. the training is conducted with a batch size of 1600, distributed over either 8× a100-80g or 32× v100-32g. ## discussion and conclusion this paper introduces evl gen , an efficient and streamlined pre-training framework for visionlanguage generative models. like blip-2, evl gen employs frozen vit and llm. it further leverages a conventional transformer architecture with tokenmerging capabilities, known as tomeformer, to act as the vision-to-language connector. compared to blip-2, evl gen offers the distinct advantage of one-stage training. this reduces computational overhead and maintains competitive performance even with only 1/3 to 1/6 of the computational budget required by blip-2. we have also extended evl gen 's applicability to video captioning tasks by incorporating the temporal attentive soft token contextualizing into its vit. this enhances the model's temporal modeling capabilities, culminating in the creation of evl gen -video. this extension has proven efficacious, delivering commendable performance even without specialized video-text pre-training. our investigation underscores that a temporal module, which does not disrupt the integration of the well-pretrained image-text model (e.g., blip-2 and evl gen ), is a key factor contributing to this success. evl gen demonstrates the possibility of achieving state-of-the-art performance in vision-language tasks without the need for complex training regimens or high computational budgets. this work thus makes a significant contribution to the ongoing efforts to develop more accessible, efficient, and powerful models for understanding and generating visual and textual information.