Efficient Vision-Language pre-training via domain-specific learning for human activities (2024)

Abstract: Current Vision-Language (VL) models owe their success to large-scale pre-training on web-collected data, which in turn requires high-capacity architectures and large compute resources for training. We posit that when the downstream tasks are known in advance, which is in practice common, the pretraining process can be aligned to the downstream domain, leading to more efficient and accurate models, while shortening the pretraining step. To this end, we introduce a domain-aligned pretraining strategy that, without additional data collection, improves the accuracy on a domain of interest, herein, that of human activities, while largely preserving the generalist knowledge. At the core of our approach stands a new LLM-based method that, provided with a simple set of concept seeds, produces a concept hierarchy with high coverage of the target domain.The concept hierarchy is used to filter a large-scale web-crawled dataset and, then, enhance the resulting instances with targeted synthetic labels. We study in depth how to train such approaches and their resulting behavior. We further show generalization to video-based data by introducing a fast adaptation approach for transitioning from a static (image) model to a dynamic one (i.e. with temporal modeling). On the domain of interest, our approach significantly outperforms models trained on up to 60\times more samples and between 10-100\times shorter training schedules for image retrieval, video retrieval and action recognition. Code will be released.

## introduction billion-scale vision-language pre-training on web collected image-text data @xcite @xcite @xcite @xcite has significantly pushed the state-of-theart for both uni-modal (e.g. action recognition) * -denotes equal contribution and multi-modal understanding (e.g. image captioning, vqa, etc.). the current trend in vl pretraining continues to scale up both the training datasets @xcite and model sizes @xcite , further increasing the already very large training-and test-time computational requirements. this is in part due to the generalist nature of such models. however, the domain of application is often known in practice. ideally, we should use this prior knowledge to increase the training and inference time efficiency. this is the very goal of our work, as we propose a novel methodology to align the vl pre-training process to a given domain, without using additional data, and without catastrophically compromising the generalist capabilities. to the best of our knowledge, this is the very first work to systematically tackle and study such a setting. to illustrate these ideas and due to its relevance in many important tasks/applications, we focus on pre-training specialized to the domain of human activities. with this in mind, we introduce a new llmbased method that, provided with a simple set of concept seeds, iteratively produces a hierarchy of textual queries that provide high coverage of the concepts included in the target domain. these textual queries, alongside a set of predefined safety and quality filters, are used to retrieve relevant data from an internet-scale web-crawled dataset. we further enhance the retrieved instances using an image captioning model. this ensures the relevance of the textual captions, which is particularly important when relying on text-to-image retrieval. empirically, we also show that training with domainfocused data naturally increases the frequency of hard negatives within a given batch, improving the training efficiency. this is a notable difference with other works within this area of vl model improvement via data enhancement, such as @xcite ) that focus on domain-invariant processes, using hardcoded rules (e.g. based on wordnet, sentence complexity, etc.). in summary, our contributions are: vision-language pre-training has emerged as the foremost approach for representation learning and training of robust zero-shot models. based on the training objective, methods can be broadly placed into two categories: generative @xcite @xcite and contrastive @xcite @xcite @xcite @xcite , although, recently promising results were shown by approaches situated at their intersection @xcite @xcite . from an architectural and training point of view, our approach closely follows clip @xcite and hence it is part of the contrastive family of models. such models are trained using a contrastive loss with the goal of learning a joint embedding space for the two modalities (i.e. vision and language). following clip @xcite subsequent works seek to either improve the loss function @xcite @xcite @xcite @xcite @xcite @xcite or improve and increase the model size @xcite @xcite and/or the dataset @xcite @xcite used. for example, declip @xcite introduces multiview and nearest-neighbor supervision, filip @xcite applies the contrastive loss in a finegrained manner while siglip @xcite replaces the contrastive loss with a sigmoid one. hiclip @xcite introduces hierarchical attention, while @xcite improves the architecture and the training scheduler. in contrast to the aforementioned works, we do not change the image-based architecture nor the training objective, focusing instead on domain-specific vl pre-training for obtaining highly discriminative and robust representations in a data-, training-and compute-efficient manner. dataset construction has very recently attracted the interest of the community that started to transition from huge noisy datasets (e.g.: laion-400 @xcite , laion-5b (schuh-mann et al., 2022), align @xcite , flamingo @xcite , siglip @xcite ) to cleaner, higher quality ones @xcite . metaclip @xcite attempts to recreate the collection process from the closed source dataset of @xcite by querying based on wikipedia articles, bigrams, and wordnet data combined with a filtering and balancing process. diht @xcite introduces a rule-based system for retaining higher quality samples, while @xcite performs data deduplication. building on prior works, datacomp @xcite provides a unified framework that provides rule-based recipes for constructing datasets ranging in size from 12.8m to 12.8b. however, none of these approaches consider the case of domain-specific training. moreover, they require manual/hand-crafted seeding of the data filtering, as opposed to our llm-based process. we significantly outperform all these methods while being up to an order of magnitude more data-and training-efficient. in the section, we introduce hc-vl, a new vl model trained with domain priors that significantly improves the in-domain performance without notable out-domain degradations, while requiring (a) no additional data, (b) up to 60× fewer training samples and (c) between 10 -100× fewer training iterations. this model builds around our newly introduced llm-based data filtering and enhancement strategy, presented in sec. 3.1. the model training itself, and the arch. changes made for fast adaptation to temporal data, are detailed in sec. 3.2. ## comparison with state-of-the-art to showcase the effectiveness of the proposed domain-specific pre-training, we evaluate our models for image retrieval, video retrieval and action recognition as they are tasks well aligned with the domain of human activities. see the appendix for pre-training details and additional evaluations. specifically, we compare hc-vl and hc-vl+ with all the equivalently sized models available as part of the openclip repo @xcite , noting that not all methods offer both the b/32 and b/16 variants. moreover, we emphasize that all models we compare with are trained using longer schedulers on a significantly larger number of samples, ranging from 400m for openclip (laion-400) @xcite to 10b for siglip @xcite (see fig. 1 ). in comparison, our models are trained only on 167m samples. ## ablation studies effect of data quantity: our models are trained on significantly fewer samples (167m) compared with the current state-of-the-art vl models, commonly trained on 400m @xcite , 2b @xcite or even 6.6b @xcite samples. going one step further, herein, we explore the performance of our model in even lower data regimes. in tab. 6, we report ## conclusions in this work, we introduced a new take on vl pretraining, that aims to take advantage of in-domain priors without degrading the generalizability of the model, herein tested for the domain of human activities. to this end, we proposed a new llm-based method for automatic data design which enables the construction of a vast human activity-specific dataset by retrieving, filtering, and re-captioning related samples from laion, on which we train our vl model, hc-vl. furthermore, we introduced a set of new architectural changes that allow for the fast addition of temporal modeling, without compromising the generalization capabilities of our image-based model. our models achieve state-ofthe-art performance across a large suite of datasets for zero-shot image recognition, video recognition and zero-shot action recognition. notably, this is achieved without using additional data and while being up to 60× more data efficient and 10 -100× faster to train due to requiring fewer iterations.