D is C o CLIP : A Distributional Compositional Tensor Network Encoder for Vision-Language Understanding (2025)

Abstract: Recent vision–language models excel at large-scale image–text alignment but often neglect the compositional structure of language, leading to failures on tasks that hinge on word order and predicate–argument structure. We introduce DisCoCLIP, a multimodal encoder that combines a frozen CLIP vision transformer with a novel tensor network text encoder that explicitly encodes syntactic structure. Sentences are parsed with a Combinatory Categorial Grammar parser to yield distributional word tensors whose contractions mirror the sentence’s grammatical derivation. To keep the model efficient, high-order tensors are factorized with tensor decompositions, reducing parameter count from tens of millions to under one million. Trained end-to-end with a self-supervised contrastive loss, DisCoCLIP markedly improves sensitivity to verb semantics and word order: it raises CLIP’s SVO-Probes verb accuracy from 77.6% to 82.4%, boosts ARO attribution and relation scores by over 9% and 4%, and achieves 93.7% on a newly introduced SVO-Swap benchmark. These results demonstrate that embedding explicit linguistic structure via tensor networks yields interpretable, parameter-efficient representations that substantially improve compositional reasoning in vision–language tasks.

## introduction vision-language understanding is a key challenge in ai, with applications to image captioning and multimodal retrieval. models like openai's clip @xcite have shown that largescale joint embeddings can effectively connect visual and textual data. however, these models mainly rely on transformer architectures with dense attention, which may overlook the linguistic structure. for instance, recent evaluations of cliplike models show that they often ignore word order, acting like bags-of-words @xcite @xcite . the attribution, relation and order (aro) benchmark @xcite checks if they are able to understand the correct word order. similarly, the svoprobes benchmark (hendricks and @xcite tests if these models mainly focus on nouns, or are also able to recognise verbs. both of these issues have been common challenges for visionlanguage models. it has been argued that these challenges stem from clip-like models being trained on websourced image-caption pairs, where captions (often alt-texts) frequently ignore word order and verb usage. as a result, their contrastive learning is not sensitive to linguistic structure @xcite . while training with hard negatives could address this, such samples are costly to source. instead, we introduce discoclip, the first model for vision and language with a text encoder that fully incorporates the compositional linguistic structure of text with the distributions of the words therein. to achieve this, we represent sentences as tensor networks, where each word is encoded as a tensor and interactions between words are captured through a series of tensor contractions. the advantages of using a tensor network text encoder are twofold. first, it enables explicit encoding of both syntactic structure and statistical semantic information, making the resulting text representations more interpretable than those produced by transformer-based encoders. second, tensor network decompositions can dramatically reduce the number of parameters required, allowing for efficient modelling of high-order interactions without incurring exponential growth in tensor size. tensor networks are widely used in quantum machine learning to capture higher order data correlations @xcite @xcite . their use in vision-language tasks might lead to further advantages coming from the quan-tum world. discoclip was evaluated on two existing benchmarks on compositional capability: svo-probes and aro, as well as on a new svo-swap benchmark created by swapping subjects and objects. we compare the performance of discoclip with clip, openclip @xcite and blip @xcite on these benchmarks. discoclip outperforms clip and openclip on verb understanding by 4.82% and 1.01%. it also outperforms clip in overall performance by 1.3%, but falls behind openclip and blip by 2.05% and 7.9%. on svo-swap, it achieves an accuracy of 93.68% outperforming all three of clip, openclip and blip by a large margin (30.52% -57.04%). on aro-relation, again it outperforms all three of the clip models by 4.28% -5.1%, in aro-attribution, it outperforms clip and openclip by 9.01% and 10.88%, but falls behind blip by 8.45%. in summary, discoclip achieves comparable performance to transformer-based models with orders of magnitude fewer parameters. the use of tensor decomposition enables efficient representation and computation, making our model more parameter-efficient and potentially more robust when training data is limited. to our knowledge, it is the first time that the theory of tensor networks has been used to model the structure of language or used in vision-language tasks. our work provides a new witness for the applications of tensor networks to machine learning and further showcases the advantage of using them. ## related work several approaches have been proposed to address these challenges in vision-language models. some incorporate aspects of linguistic structure @xcite , others introduce hard negatives @xcite , and some incentivize learning by explicitly rewarding the model for capturing linguistic elements such as adjectives and verbs @xcite . tensor networks were introduced to make the numerical treatment of many-body quantum states feasible by exploiting their internal structure @xcite . such states naturally live in exponentially large tensor-product spaces, which are difficult to handle directly. a tensor network circumvents this by factorizing a single, high-order tensor into a set of lower-order tensors, whose indices are glued together by contraction operations. in a tensor train (also known as a matrix product state, or mps), these tensors are arranged in a strictly onedimensional sequence, with each tensor contracted only to its immediate predecessor and successor through shared bond indices; by contrast, a tree tensor network connects tensors in a branching, hierarchical structure. tensor networks have found applications outside physics, especially in machine learning where they are used for sequence modelling @xcite , optimizing the computations of neural networks @xcite , and in general any large-scale optimization problem @xcite , such as latent feature extraction (stoudenmire, 2018) and security @xcite . their decomposition methods have been tested on image classification tasks @xcite @xcite , word statistics, and document retrieval from large corpora of text @xcite @xcite . tensors and the contraction operation between them were also used in a model of meaning known as "compositional distributional semantics" @xcite @xcite . in this model, the meaning of each word is either a vector or a higher-order tensor. the orders of the tensors are determined by the grammatical roles of words. meanings of nouns are vectors, where as meanings of words with functional roles such as adjectives and verbs are matrices and cubes. discoclip is inspired by compositional distributional semantics and the theory of tensor networks. we denote the meaning of a piece of text by a tensor network. in this tensor network, the tensors encode meanings of words, the layout of the tensor network represents the syntactic structure of the sentence. other tensor network layouts are used as baselines to test how useful is encoding less structure, such as word order and bags-of-words. another key novelty of our model is that it extends compositional distributional semantics to a multimodal setting. previous multimodal adaptations include @xcite for compositional concept learning, @xcite for audio-text retrieval, and @xcite for verb understanding in clip. however, discoclip differs from these approaches in two important ways. first, our model is more general: it handles sentences of arbitrary syntactic structure, whereas prior work typically focuses on specific constructions such as subject-verb-object @xcite or adjective-noun pairs @xcite . second, disco-clip features an end-to-end pipeline trained with a single objective function, in contrast to previous methods that require a separate objective for their different text and audio/image model components. this unified approach enables more flexible and scalable multimodal learning. ## basics of tensor networks a tensor network is a collection of tensors contracted together to form a new tensor. an ordern tensor t is a multi-dimensional array @xmath0, where d i is the dimension of the i-th index. elements are denoted by t i 1 ,...,in , with each i k ranging from 0 to d k-1 . scalars, vectors, and matrices are tensors of order 0, 1, and 2, respectively. tensor contractions. tensors can be multiplied together by contracting over a shared index, which generalizes matrix multiplication. for example, given two tensors @xmath1, their contraction over the second index yields a new scalar vector matrix order-3 tensor dot-product matrix-vector bilinear form figure 2: graphical representation of tensor networks. a tensor is depicted as a node with one edge for each index of the tensor. for example a scalar has no edge, a vector has one edge, a matrix has two edges and an order-3 tensor has 3 edges. an edge of a node can be connected to another edge of another node, forming a contraction, which is a generalised form of matrix multiplication. this operation extends naturally to higher-order tensors by summing over any shared index. ## methodology our main contribution is to replace clip's transformer-based text encoder with a tensor network encoder, resulting in a new vision-language model we call discoclip. in discoclip, the text encoder constructs sentence embeddings using tensor networks that explicitly encode linguistic structure, while the image encoder remains the original clip vision transformer. by varying the layout of the tensor network, we can control the level of syntactic and semantic information captured in the text representation. given an image-caption pair, discoclip processes them in the following steps (see figure 1 ): combinatory categorial grammar (ccg) is a highly expressive formalism for modeling natural language syntax and semantics. in ccg, each word is assigned a syntactic category that reflects both its grammatical role and its combinatory potential. categories are either atomic (such as noun phrase n p or sentence s) or functional, where functional categories specify how a word combines with its arguments. functional types take the form y /x or y \x, indicating that the word expects an argument of type x to its right (/) or left (\), and yields a result of type y . for example, adjectives have type n p/n p , intransitive verbs have type s\n p , and transitive verbs have type (s\n p )/n p . the combinatory rules of ccg allow for the composition of these categories to cancel out the functional types and yield a sentence s type. the two main rules are forward application (>) and backward application (<): other ccg rules include forward and backward composition, which are used to combine auxiliary verbs with their arguments, and forward and backward cross-composition, used to combine categories with long distance dependencies such as gapping. another notable ccg rule is type-raising, which enables specific combinations of categories, e.g. from left to right. this feature helps the ccg align with psycholinguistic theories. for instance, in english, it will allow categories to combine from left to right and form incremental parses that support theories of human sentence processing. a distributional compositional (disco) semantics has been developed for ccg @xcite @xcite . this semantics assigns to a word w with a ccg category composed of n atomic categories a multilinear map f w with n arguments each v i is a finite-dimensional vector space over the field of reals r. equivalently, f w can be represented by a tensor of in the space here, each atomic type corresponds to an index of the tensor. for example, a noun with the type n p is assigned a vector (order-1 tensor), while an adjective with the type n p/n p is assigned a linear map that takes a vector and returns a vector, which can be represented as a matrix (order-2 tensor). a transitive verb with the type (s\n p )/n p is assigned a bilinear map that takes two vectors and returns another vector, i.e. a cube (an order-3 tensor), and so on. for the general formulae of these representations, see @xcite and @xcite . given the ccg parse tree, the word tensors are composed by performing tensor contractions that mirror the syntactic reductions specified by the tree. each time a combinatory rule (such as forward or backward application) is applied in the parse, the corresponding word tensors are contracted along the appropriate indices. this process recursively combines the tensors according to the grammatical structure, ultimately yielding a single vector representation for the entire sentence. such semantics was developed in @xcite and leads to the compact tensor network structure. an alternative semantics presented in @xcite assigns to every word a vector and models the grammatical compositions (represented by ccg rules such as forward and backward application) by a shared order-3 tensor. this tensor acts as a universal composition operator of all compositional operators. this approach yields the tree tensor network structure, where the parse tree topology is preserved but all internal nodes use the same composition tensor to combine their child representations. ## contrastive learning we train discoclip using contrastive learning, where the image encoder f (frozen clip) and the tensor network text encoder g map image-caption pairs (x, y) to embeddings (x, y). the goal is to bring true (positive) pairs closer and push (negative) mismatched pairs apart in the joint embedding space. for a batch of b positive pairs, all non-matching image-caption combinations in the batch (b(b -1)) serve as negatives, following the in-batch negative sampling of clip @xcite . these are considered easy negatives, as opposed to more challenging, hand-crafted hard negatives. we use the widely adopted infonce loss @xcite to train the model. given a batch of b image-caption pairs with embeddings the loss thus encourages higher similarity for true pairs and lower for mismatched ones. we evaluate our approach on two key benchmarks for vision-language understanding: svo-probes and aro. the svo-probes dataset is designed to test whether models can distinguish fine-grained changes in the image which corresponds to variations in subject, verb, or object. the task is to determine which of the two images correctly matches a given caption. in contrast, the aro (attribution, relation, and order) dataset assesses a model's ability to correctly compose meanings in a sentence. the task is to determine which of the two captions correctly describes a given image. together, these datasets provide a comprehensive evaluation of both compositional and structural language understanding in multimodal models. ## conclusion in this work, we introduced discoclip, a vision-language model that replaces the standard transformer-based text encoder with a structureinformed tensor network. by leveraging the compositional layouts of tensor networks inspired by compositional distributional semantics and quantum-inspired tensor decompositions, our approach explicitly encodes linguistic structure and achieves competitive performance on challenging multimodal benchmarks such as svo-probes and aro. our experiments demonstrate that structure-aware tensor networks, particularly the compact model that was a dense variant of the syntactic parse tree, can match or surpass classical neural models in tasks requiring fine-grained understanding of sentence structure. our model also uses significantly fewer number of parameters in comparison to transformer-based models such as clip. these results highlight the potential of tensor network architectures as interpretable and parameterefficient alternatives for multimodal learning. future work will explore scaling these models to larger datasets, working with complex datasets such as winoground @xcite , exploring the quantum connections and training circuit ansatze, and extending the approach to more complex linguistic phenomena.