V erb CLIP : Improving Verb Understanding in Vision-Language Models with Compositional Structures (2024)

Abstract: Verbs describe the dynamics of interactions between people, objects, and their environments. They play a crucial role in language formation and understanding. Nonetheless, recent vision-language models like CLIP predominantly rely on nouns and have a limited account of verbs. This limitation affects their performance in tasks requiring action recognition and scene understanding. In this work, we introduce VerbCLIP, a verb-centric vision-language model which learns meanings of verbs based on a compositional approach to statistical machine learning. Our methods significantly outperform CLIP in zero-shot performance on the VALSE, VL-Checklist, and SVO-Probes datasets, with improvements of +2.38%, +3.14%, and +1.47%, without fine-tuning. Fine-tuning resulted in further improvements, with gains of +2.85% and +9.2% on the VALSE and VL-Checklist datasets.

## introduction trained on extensive datasets of image-caption pairs, current vision-and-language models (vlms) excel in various applications, yet stall in tasks that require structural knowledge and compositional reasoning @xcite . research by @xcite demonstrates some of the difficulties they face in understanding attributes, relationships, and order information. more specifically, @xcite points out that vlms often fail to distinguish between different verbs, instead relying predominantly on noun understanding. one possible reason for this issue is the inherent biases within the training datasets. these datasets host a limited number of examples where captions share similar contexts but differ in verbs. as a result, they focus on specific objects and subjects, with minimal emphasis on verbs. this tendency is a form of shortcut learning, a phenomenon in deep neural networks where models opt for simpler, superficial solutions over a deeper understanding @xcite . conversely, compositional distributional semantic models (cdsms) @xcite @xcite learn meaning representations of sentences by considering their compositional linguistic structures, such as the relationships between verbs and their subjects and objects. in the model proposed by @xcite , verbs are represented as tensors that take lower-order word representations, typically vectors, as arguments. this means that intransitive verbs are represented as matrices, transitive verbs as cubes, and ditransitive verbs as hypercubes. these tensor-based representations have shown promising results in tasks such as verb disambiguation and sentence similarity @xcite . cdsms have primarily been applied to text-only data and tasks, and have recently been used as text encoders for clip @xcite . the novel contribution of this paper lies in integrating vlms with cdsms within a framework called verbclip to enhance verb understanding. we implement various methods for learning verb tensors on an image-caption matching task and evaluate these methods on valse, vl-checklist, and svo-probes datasets. our best tensor learning method achieves improvements of +2.38%, +3.14%, and +1.47% over clip. beyond these quantitative improvements, a significant advantage of verbclip is that it does not require training from scratch. our code and data are available at https://github.com/lanlos-lab/verbclip . ## methodology we present an overview of our framework, illustrated in figure 1 . it utilises frozen clip as the backbone. initially, we input the original sentence and image into clip's encoders to obtain a similarity score, reflecting the overall alignment between the general semantics of the text and the image. simultaneously, we extract the subject-verb-object triplet from the sentence. these components are encoded separately: the subject and object as vectors, and the verb as matrices, forming a compositional text embedding that captures the detailed semantic relationships. we then calculate a similarity score between the compositional text embedding and the image embedding. we add the two scores to produce the final matching score. ## experiment we focus on the task of matching images with correct captions. an image is described by both a positive and a negative caption; the negative caption differs from the positive only by a verb. our aim is to achieve a higher matching score between the image and the positive caption compared to the negative one. evaluation datasets we test our methods on valse @xcite , vl-checklist @xcite , and svo-probes (hendricks and @xcite . detailed descriptions of the datasets are in the above papers; however, for this study, we selected only those entries where the verb differs between the positive and negative captions, while the subjects and objects are the same. for the svo-probes, we create negative captions by substituting the verb in the positive caption with its corresponding negative form from the given negative (svo) triplet. for example, given a positive caption 'a woman is running in the field' and a ## results and discussion the compositional tensor-based methods significantly outperform clip and vector-based models, with copy-add showing the highest perfor-mance in most cases. copy-add appears capable of utilising information from the combination of subject-verb and verb-object, and incorporating further information from the object and subject. this highlights the importance of ordering and syntactic information in the compositional methods. upon fine-tuning the weights, α and β, we noticed further improvement (+2.85% and +9.2% on the valse and vl-checklist datasets respectively). we noticed lower performance improvements on the svo-probes dataset compared to valse and vl-checklist. this discrepancy is likely due to the nature of the svo-probes dataset, which contains sketchy samples and tends to be noisy, with significant issues such as object mismatches, as detailed in @xcite . in terms of learning verb matrices, regression methods demonstrated lower accuracies, whereas the kronecker (kron) and relational (rel) methods performed better. the fact that kron requires no training data makes it an effortless choice for constructing verb matrices, while still providing competitive performance. in terms of verb-type performance, the copy-add model significantly improved accuracy for interaction-based verbs such as "hang" (+12.5%), "hold" (+11.6%), "attached" (+3.7%), and "take" (+29.62%). however, while it struggled with some visually static verbs like "stand" (-5.8%) and "sit" (-6.0%), it showed improvement in others such as "observe" (+50%) and "look" (+10.87%). furthermore, we tested sentence pairs where the subject and object nouns are swapped, such as "a man lies on the sofa" vs "a sofa lies on the man". clip often misinterprets these as equally plausible, reflecting its approach of processing text as independent words, similar to a bag-of-words approach. in contrast, copy-add model correctly identifies "a man lies on the sofa" as the correct caption by capturing structured detailed semantics. overall, verbclip a baby speaks on the telephone. ## limitations creating verb matrices or tensors is computationally intensive, which poses a significant challenge when scaling to very large pretraining datasets. additionally, our approach assumes a fixed linguistic structure, typically the subject-verb-object format, which does not account for the varied and flexible ways verbs are used in natural language. however, tensors are natural components of quantum systems, and quantum computing resources can efficiently learn them. the quantum natural language processing (qnlp) framework @xcite , inspired by categorical quantum mechanics and the dis-cocat (distributional compositional categorical) framework, uses string diagrams to translate grammatical structures into quantum processes. this advanced option could offer a promising solution. ## conclusion the clip model is noted for its limited ability to understand verbs, often relying heavily on nouns. our approach seeks to mitigate this issue by introducing verb-focused compositional methods, which have demonstrated enhanced performance across the svo-probes, vl-checklist and valse datasets. our framework can boost the zero-shot inference capability of other models, such as slip @xcite and blip @xcite , without the need for further training or fine-tuning. scaling to longer and more complicated sentences with varied grammatical structures is a work in progress.