U ni F ine: A Unified and Fine-grained Approach for Zero-shot Vision-Language Understanding (2023)

Abstract: Vision-language tasks, such as VQA, SNLI-VE, and VCR are challenging because they require the model’s reasoning ability to understand the semantics of the visual world and natural language. Supervised methods working for vision-language tasks have been well-studied. However, solving these tasks in a zero-shot setting is less explored. Since Contrastive Language-Image Pre-training (CLIP) has shown remarkable zero-shot performance on image-text matching, previous works utilized its strong zero-shot ability by converting vision-language tasks into an image-text matching problem, and they mainly consider global-level matching (e.g., the whole image or sentence). However, we find visual and textual fine-grained information, e.g., keywords in the sentence and objects in the image, can be fairly informative for semantics understanding. Inspired by this, we propose a unified framework to take advantage of the fine-grained information for zero-shot vision-language learning, covering multiple tasks such as VQA, SNLI-VE, and VCR. Our experiments show that our framework outperforms former zero-shot methods on VQA and achieves substantial improvement on SNLI-VE and VCR. Furthermore, our ablation studies confirm the effectiveness and generalizability of our proposed method.

## introduction vqa @xcite , snli-ve @xcite , and vcr @xcite are visionlanguage tasks, which utilize the text and corresponding image to test a system's cross-modal reasoning ability. these tasks are challenging as requiring models to obtain a joint understanding of visual and textual modality. nevertheless, they are also meaningful since this capability plays an essential role in daily human-robot interaction, e.g., asking a robot how many people are in the image. despite the difficulty, a line of work @xcite @xcite @xcite has been dedicated to resolving these vision-language tasks in a supervised setting and obtaining impressive progress. however, these methods all suffer from a significant problem of being costly as they require expert knowledge to collect well-annotated imagetext data. on the other hand, zero-shot methods for vision-language tasks can successfully bypass this problem without costly annotations. unfortunately, limited methods and relatively fewer works have been dedicated to exploring this direction. recently, clip @xcite has been proposed to acquire visual concepts using natural language supervision. it jointly trains an image encoder and a text encoder on 400m noisy imagetext pairs collected from the internet by aligning images and texts through a contrastive loss. previous works @xcite @xcite demonstrated that clip can achieve strong zeroshot performance of vision-language tasks by converting original tasks into the image-text matching format. however, they mainly consider matching on an instance or global level, i.e., the whole image or sentence, ignoring the significance of finegrained elements, e.g., keywords in the sentence and objects in the image. meanwhile, we find these fine-grained elements are important for specific downstream tasks, especially in zero-shot learning. for instance, in fig. 1 , clip makes three incorrect predictions in three zero-shot vision-language tasks. for vqa, the model infers the wrong object "pancake" for the verb "eating", as it does not capture the details in the image (pizza on the table) and captions (pizza is mentioned). we posit that if we can find a proper solution to navigate the model to focus on these detailed pieces of textual and visual information, the model would likely have a better chance of selecting the correct answer label. this conjecture also seems true and generalizable across multiple zero-shot downstream tasks as shown by the three examples from different vision-language tasks, i.e., vcr, vqa, and snli-ve in fig. 1 . yet, we also recognize potential challenges also exist as those different tasks may differ from many perspectives including the distribution of image categories or scenes, the different semantic focus, and format of text premises between declarative statements and questions, and different task formats in terms of image-text matching or classification. to overcome these challenges, we first identify two common fundamental steps required to utilize the fine-grained information across different visionlanguage tasks: 1) extraction of the fine-grained information from context information, e.g., the extraction of the word "pizza" from the caption in vqa as in fig. 1 . 2) the semantic matching between these extracted fine-grained information and answer choices or hypothesis. based on these, we propose a unified approach leveraging these two common steps thus it can assist the model to generalize over different vision-language tasks. for the extractor, we have two branches -1) the vision branch and 2) the textual branch. in the vision branch, we employ faster-rcnn @xcite to extract object-level information. we select relevant object regions guided by the question in vqa and vcr or hypothesis in snli-ve. after that, we concatenate the whole image and its selected image regions and input them into the image encoder of clip. for textual information extraction, we exploit rich information from the image caption generated by a recently-developed captioning model ofa @xcite and question in vqa and vcr or hypothesis in snli-ve to boost the zero-shot performance. it's noted that although we employ the image caption and question on a sentence level rather than a word level, we compute the cosine similarity between them and answer texts, which means if there are keywords in the answer texts which can be matched in the caption or question, then we will obtain high scores in zero-shot prediction. therefore, it is still a process of fine-grained information extraction. by using fine-grained information, our model outperforms previous methods on zero-shot vqa and we are the first to benchmark zero-shot vcr and snli-ve. the experiments confirm the effectiveness of our proposed method. our contributions can be summarized as follows: vision-language pre-trained models. early vision-language pre-trained models @xcite @xcite @xcite utilize cross-modal transformer @xcite pretrained on well-annotated image-text pairs. different from these models, contrastive learning frameworks @xcite @xcite are trained on noisy image-text pairs crawled from the internet through contrastive loss, which employs the dot product between visual and textual modality. due to the large-scale training data, these models acquire rich prior knowledge and show strong zero-shot ability on vision benchmarks like imagenet @xcite . vision-language zero-shot learning. there is a line of work utilizing clip to do zero-shot learning for vision-language tasks. reclip @xcite utilizes clip to present a zero-shot method for referring expression comprehension (rec), which outperforms prior zero-shot rec approaches. clip-vil @xcite exploits clip to do zero-shot vqa by simply concatenating question and answer pair for each question and constructing "question: [question text] answer: [answer text]" as the prompt. then, they feed the text and image into the text encoder and the image encoder of clip, which produces the near-chance clip-te the color of the flowers is yellow. the color of the flowers is yellow and white. the color of the flowers is white. answers answer chosen clip-ve image vqa, vcr the flowers are on a table. level performance. the most relevant work to ours is tapc @xcite , which manually designs the prompt and leverages t5 @xcite , a large pre-trained text-to-text transformer, to convert the question-answering problem into the image-text matching task. then, it employs clip's remarkable zero-shot image-text matching ability on vqa, whose results surpass clip-vil by a large margin. however, these works handle different tasks on an instance level rather than fully utilizing the visual and textual fine-grained information (i.e., keywords in the sentence and objects in the image) like ours. moreover, we can tackle a diverse set of tasks but they just concentrate on one specific task. ## method in this section, we introduce our method for visual and textual fine-grained information extraction to improve zero-shot learning of vision-language tasks including vqa, vcr, and snli-ve. ## conclusion in this work, we propose a unified and fine-grained approach for vision-language tasks including vqa, snli-ve, and vcr. we outperform previous clip-based methods for zero-shot vqa. plus, we are the first to empirically study clip's zero-shot ability for snli-ve and vcr, which achieves strong zero-shot performance. in addition to the benchmark comparison, we conduct extensive ablation studies confirming the significance of visual and textual fine-grained information and the generalizability of our proposed method.