RWKV - CLIP : A Robust Vision-Language Representation Learner (2024)

Abstract: Contrastive Language-Image Pre-training (CLIP) has significantly improved performance in various vision-language tasks by expanding the dataset with image-text pairs obtained from the web. This paper further explores CLIP from the perspectives of data and model architecture. To mitigate the impact of the noise data and enhance the quality of large-scale image-text data crawled from the internet, we introduce a diverse description generation framework that can leverage Large Language Models (LLMs) to combine and refine information from web-based image-text pairs, synthetic captions, and detection tags. Additionally, we propose RWKV-CLIP, the first RWKV-driven vision-language representation learning model that combines the effective parallel training of transformers with the efficient inference of RNNs. Extensive experiments across different model scales and pre-training datasets demonstrate that RWKV-CLIP is a robust vision-language representation learner and it achieves state-of-the-art performance across multiple downstream tasks, including linear probing, zero-shot classification, and zero-shot image-text retrieval. To facilitate future research, the code and pre-trained models are released at https://github.com/deepglint/RWKV-CLIP.

## introduction the proliferation of mobile networks and social platforms has greatly accelerated the large-scale production of image-text pairs @xcite . this unprecedented abundance of data has established the foundation for visionlanguage pre-training. contrastive language-image pre-training (clip) employs two distinct unimodal encoders for images and text, utilizing a contrastive loss, a highly effective mechanism for representation learning. having been pre-trained on extensive image-text pairs collected from the internet, clip demonstrates strong transferability and has been widely applied across various domains @xcite . in recent years, many large-scale image-text datasets collected from the internet have been released. laion400m @xcite is created for research purposes and it contains 400 million image-text pairs curated using the clip model. building on this, laion5b @xcite , which consists of 5.85 billion clipfiltered image-text pairs, successfully replicates and fine-tunes basic models such as clip. however, using the clip model to filter web-based image-text pairs still retains a considerable presence of noisy data. to improve data quality, @xcite employs various strategies such as basic filtering, clip score filtering, and text&image-based filtering. however, inherent characteristics of internet data, such as abstract text representations and semantic discrepancies between text and images, remain significant obstacles. in recent years, the transformer @xcite model has been extensively applied in largescale representation learning, yielding significant performance improvements across multiple downstream tasks @xcite @xcite , including image classification @xcite ), text generation @xcite , and speech recognition @xcite . despite these achievements, the quadratic computational complexity inherent in transformer limits its capacity to effectively process high-resolution images and long sequences, posing a substantial challenge to its broader applicability across varied domains. in this paper, we design a framework for generating diverse descriptions. following alip @xcite , we first use the ofa @xcite model to generate synthetic descriptions consistent with image content. however, constrained by the training data, ofa can only partially identify coarse-grained object categories. therefore, we introduce an open-set image tagging model ram++ @xcite to capture more detailed and precise semantic information from images. by leveraging llms, we synthesize and refine information from web-based texts, synthetic captions, and detection tags. additionally, inspired by rwkv @xcite and vision-rwkv @xcite , we propose rwkv-clip, the first rwkv-driven vision-language representation learning model that combines the effective parallel training of transformers with the efficient inference of rnns. extensive experiments across various model scales and pre-training datasets demonstrate that rwkv-clip is a robust and efficient vision-language representation learner. the main contributions of this paper are summarized as follows: ## method in this section, we first introduce a diverse description generation framework that leverages the capabilities of large language models to integrate information from web-based texts, synthetic captions, and detection tags. subsequently, we provide a detailed exposition of rwkv-clip. ## conclusion in this paper, we further explore clip from the perspectives of data and model architecture. we introduce a diverse description generation framework that can leverage large language models (llms) to combine and refine information from web-based image-text pairs, synthetic captions, and detection tags. besides, we propose rwkv-clip, the first rwkv-driven vision-language representation learning model that combines the effective parallel training of transformers with the efficient inference of rnns. our method demonstrates superior performance across various model scales and pretraining datasets on different downstream tasks. we hope that our work provides insights into visionlanguage representation learning models.