Individuation in Neural Models with and without Visual Grounding (2024)

Abstract: We show differences between a language-and-vision model CLIP and two text-only models — FastText and SBERT — when it comes to the encoding of individuation information. We study latent representations that CLIP provides for substrates, granular aggregates, and various numbers of objects. We demonstrate that CLIP embeddings capture quantitative differences in individuation better than models trained on text-only data. Moreover, the individuation hierarchy we deduce from the CLIP embeddings agrees with the hierarchies proposed in linguistics and cognitive science.

## introduction recent results in multimodal 1 vision and language (v&l) models lead to intriguing research questions. for instance, one exciting research direction would be to search for the synergistic effects of multimodality. so far, to the best of our knowledge, no definitive finds were made on this front, despite the growing body of research on v&l model evaluation. for example, @xcite provide a benchmark to assess the visual grounding capabilities of v&l models. the authors conclude that current models have difficulty addressing most phenomena that require models to ground linguistic information in the visual modality. @xcite present a benchmark for visio-linguistic compositional reasoning and also find that none of the modern v&l models does much better than chance. this paper presents the first case of such visiolinguistic synergies. namely, it studies the phenomenon of individuation and how v&l models represent objects: how they distinguish objects from substances and how they track objects and their quantity in sets that contain more than one object. we demonstrate 1 for a detailed review of various aspects of multimodal machine learning, we address the reader to @xcite that clip's @xcite latent representations have properties that differ from those of the models that use only textual data. moreover, this emergent property seems to agree with individuation scales proposed earlier by linguists and cognition researchers. ## individuation individuation is generally understood as basic principles that guide the distinction between objects and substances, as well as the distinction between a single object and multiple objects. individuation is not limited to visual modality -it applies crossmodally to stimuli of any kind. here, we will only focus on visual individuation and its relation to the linguistic properties of corresponding words. this section summarizes the main relevant findings on individuation from cognitive science and linguistics. operationally, individuation can be probed along two axes: 1) the quantity axis; 2) the object axis. the former corresponds to distinguishing and tracking individual objects as their quantity increases. the latter is the dependency between the individual properties of an object and its permeability as an object rather than a substance. we will now overview these two aspects of individuation -first, in cognition, then in language. ## individuation assessment we suggest estimating the models' individuation 'resolution' by inferring its individuation hierarchy. the pipeline we construct here is based on publicly available data and is motivated by cognitive and linguistic experiments on individuation discussed above. we structure this section as follows. first, we describe the list of nouns and semantic features that we use in our experiments throughout this paper. we then propose a simple way to characterize individuation in a model. in short, we will look at embeddings of noun phrases describing various quantities of objects and measure distances between different quantities of the same type of object (that is, described by the same noun). while this approach does not, of course, exclude other potential analysis tools, we believe it is a simple and effective way to demonstrate that v&l models represent individuation differently and show behavior closer to human perception of individuation. we hope that further work on individuation in modern deep neural nets refines the proposed methods or proposes more elaborate ones. let us discuss our experiments in detail and then demonstrate how clip differs from contextual text embeddings (for example, sbert, reimers and gurevych 2019) and static word embeddings (say, fasttext, @xcite ) in text-only models. ## discussion the first thing that one sees is that fasttext has almost no distinguished individuation classes and lumps a variety of classes together. it doesn't give rise to a systematic individuation hierarchy that would be similar in any way to the rankings proposed in cognitive science and linguistics. if anything, the tendency is the opposite: substances turn out to be one of the most 'individuated' classes, while animals and organisms end up on the lower end of the scale. the second observation is that the hierarchies deduced from sbert and clip are similar to the individuation hierarchies proposed in cognitive and linguistic research. as discussed in section 2, the linguistic behavior of different nouns varies in ways that can be traced back to cognitive distinctions in individuation. therefore, it comes as no surprise that a text-only model attuned to semantic distinctions that show up in distributional properties of nouns can pick up the relevant contrasts. yet a careful exploration of figure 2 allows us to see where additional -visual -modality helps develop clearer individuation-related distinctions compared to the language-only model. the individuation scale produced by clip is more fine-grained. for example, sbert lumps fruits in one individuation 'cluster' with foods and nutrients and various plants, while clip distinguishes foods and nutrients from plants. this can potentially be related to the fact that images depicting food (in particular, fruit) tend to differ from images with plants in general (say, landscapes). one of the ways to quantify the resulting differences in the model's individuation is to represent data shown in figure 2 as a graph and calculate the cliques' parameters. let us connect to vertices representing a class of nouns with an edge if the pval on figure 2 is greater than five percent. this would mean that our proxy metric based on model embeddings has difficulty distinguishing the classes. now we can count maximal cliques. the more cliques we end up with, the more separate classes are distinguished by a given model. we can also calculate the average size of the cliques. the smaller this size is, the more fine-grained the individuation hierarchy induced by a given model is. the results of those calculations are presented in another interesting aspect of this is the position of animals in the induced individuation hierarchy. while sbert puts animals somewhere in the middle of the scale, clip puts them on top with the same cluster as humans. such a position goes in line with the cognitive results mentioned earlier in section 2. finally, going back to figure 1 , one could notice that clip individuation has fewer discontinuities when compared to sbert. specifically, if @xmath0, then clip almost always recognizes that i(n) < i(k) < i(j). out of thirty-six pairwise comparisons, there are three situations when this observation does not hold. with sbert, this does not hold in eight cases out of thirty-six pairwise comparisons. another thing worth noting is the standard deviation of the individuation proxy that differs significantly for all three models; see ## conclusion this paper demonstrates that clip benefits from vision-language synergy and thus effectively encodes individuation properties for distinct entities. we suggest a method to calculate whether the model captures individuation for a given class of objects in its embeddings. using this method, we infer the individuation hierarchy that several models induce on different classes of objects. thus, we demonstrate that clip embeddings capture quantitative differences in a way that is in closer agreement with the human perception of individuation. we hope that this paper stimulates further discussion on multimodality as a source for models that are aligned with human perspective and perception.