Leverage Points in Modality Shifts: Comparing Language-only and Multimodal Word Representations (2023)

Abstract: Multimodal embeddings aim to enrich the semantic information in neural representations of language compared to text-only models. While different embeddings exhibit different applicability and performance on downstream tasks, little is known about the systematic representation differences attributed to the visual modality. Our paper compares word embeddings from three vision-and-language models (CLIP, OpenCLIP and Multilingual CLIP, Radford et al. 2021; Ilharco et al. 2021; Carlsson et al. 2022) and three text-only models, with static (FastText, Bojanowski et al. 2017) as well as contextual representations (multilingual BERT Devlin et al. 2018; XLM-RoBERTa, Conneau et al. 2019). This is the first large-scale study of the effect of visual grounding on language representations, including 46 semantic parameters. We identify meaning properties and relations that characterize words whose embeddings are most affected by the inclusion of visual modality in the training data; that is, points where visual grounding turns out most important. We find that the effect of visual modality correlates most with denotational semantic properties related to concreteness, but is also detected for several specific semantic classes, as well as for valence, a sentiment-related connotational property of linguistic expressions.

## introduction linguistic representations developed by recent large pre-trained language models (lms) @xcite @xcite proved to be very useful across a variety of practical applications. this success has given a new life to the debate around extractability and quality of semantic information in representations trained solely on textual input. according to the widely supported argument, unless the textual data is grounded in a separate space (say, visual), the linguistic representations are bound to be semantically deficient (see @xcite . we aim to shed new empirical light on the discussion of grounding in computational models by comparing language-only text representations to visually informed text representations. recent advances produced empirically successful large models pre-trained on a combination of textual and visual data @xcite @xcite . while these multimodal systems have already given rise to a plethora of applications for language-and-vision (l&v) downstream tasks, there is still little work that directly compares textual representations of language-only models to those of multimodal ones (however, see @xcite @xcite . in contrast to previous related work that focuses on model evaluation with respect to specific benchmarks, we look at the impact of visual grounding from a somewhat different, non-evaluation-based perspective. we do not aim to measure the representation quality with respect to some gold standard, but compare language-only and l&v models to each other intrinsically. our goal is to identify the areas in which the contrasts between the two kinds of models tend to lie, independent of the models' fitness for specific tasks. to do so, we focus on a set of 13k word pairs and compare cosine distances within these pairs in the embedding spaces of language-only vs. l&v models. fixing the word pairs and comparing the models allows us to measure how the change in model modality stretches the embedding space, with the word pairs as indirect reference points. the pairs are characterized along 46 different semantic parameters. this information makes it possible to identify the meaning aspects for which the change in model modality matters the most. our contributions are: our results are the following: we now discuss our data, analysis and results. the dataset consists of word pairs. to collect them, we start with 1000 most frequent words in fasttext @xcite . for each of them, we take 100 closest words, by cosine distance over fasttext embeddings. this gives 1m pairs to work with. we filter this list of pairs in several ways. first, we only keep those pairs where both words are nouns, according to both nltk foot_1 and spacy foot_2 pos labels. second, we filter out pairs where one of the words is a substring of the other or where the two words have the same lemma. this helps against some fasttext artifacts. one of the goals of our filtering strategy was to balance representation quality of the words (the frequency filter) and the chance for the pair to stand in a wordnet relation (the similarity filter). this gives us a set of pairs like the following: each of the resulting pairs was characterized along a set of properties of interest, collected over a variety of available sources of human-annotated semantic information. the properties we look at come in two big blocks: 1) the ones that characterize individual words (assigned to each word in the pair); 2) the ones that characterize a semantic relation between the words in the pair. properties for individual words included: relational semantic properties included: antonyms, synonyms, same_hyponyms, same_hypernyms, hyponyms, hypernyms. the relations were implemented as boolean labels. this is the most comprehensive list of semantic parameters for which human annotations exist on a large scale. it covers both denotational and connotational aspects of meaning of both individual words and relation within pairs. connotational meanings are represented with three sentiment-related meaning aspects only, as these are the only ones represented in a large human-annotated dataset (mohammad, 2018). additionally, word count based on wikipedia (accessed via textacy) is included for each word in all pairs as a non-semantic baseline parameter. we leave only those word pairs for which all the above mentioned parameters are defined. this gives us 13k word pairs in total, each of the pairs gets characterized along 30 individual semantic parameters (*2, for the first and the second noun in the pair) and 16 relational parameters; plus, word count for each of the words in the pair. we collect the distances between the words in each pair for their embeddings from the models of interest. as text-only models, we use fasttext @xcite and two contextualized embedding models: multilingual bert (mbert, @xcite and xlm-roberta (xlmr, @xcite . for each contextualized model, we extract three kinds of word type embeddings known to show systematic differences @xcite ; average of all token embeddings, including separator tokens, from the final encoding layer of a word presented in isolation (iso); the average encoding over the bottom 6 layers across a sample of 10 usage contexts (avg-bottom), amd the average encoding from the final layer across a sample of 10 usage contexts (avg-last). as multimodal models, we use clip, openclip and multilingual clip @xcite @xcite . for each multimodal model, we extract two different types of word type embeddings, one by encoding the word in isolation and one by averaging over sentence embeddings of 10 usage examples. the goal is to find a common ground of different models depending on their modality. in this way we hope to be able to distinguish between modelspecific idiosyncrasies and general properties of text-based representations. ## analysis we run a series of regression analyses with semantic features and relations as predictors, along with word frequency as baseline. we analyze the shift in distances within word pairs between two embedding models. to measure it, we rank all word pairs in our dataset by the ratio between the cosine distance values of the pair in the two embedding models. using ratios and ranks rather than absolute differences serves as a normalization strategy because the vector spaces have significantly different structures (see appendix a). the resulting rank of the pair is then used as the dependent variable in a regression analysis. the baseline regression model includes as predictors word frequencies in the wikipedia corpus and concreteness scores from the ghent concreteness norms dataset @xcite . to estimate the contribution of different groups of semantic features, we add them to the regression as additional predictors. this is done separately for 1. taxonomic features of the two words formalized as their wordnet supersenses @xcite ; 2. sentiment/connotation-related features of the two words extracted from nrc vad (mohammad, 2018); 3. relation within the word pair according to princeton wordnet @xcite ; 4. relation within the word pair according to con-ceptnet @xcite . all numeric parameters (concreteness scores, word frequencies, and vad values) were normalized by converting numeric values into ranks. to calculate regression, we used a standard implementation of ordinary least squares regression from the statsmodels python package. we compute adjusted r-squared values to avoid a bias from the different numbers of parameters. each fitted regression showed high significance (@xmath0.0001). ## results the results of regression analysis for several models are illustrated in figure 1 : comparing semantic features' contributions to contrasts between text models vs. other text models, on the one hand, and text models vs. l&v models, on the other hand. explanatory contributions of concreteness, vad valence and wordnet supersense 'is possession' are sensitive to model modality, unlike supersense 'is attribute'. (here and in appendix b, whiskers in the boxplots are set to 0.5 iqr.) figure 1 illustrates the effect of specific features: concreteness, valence and possession wordnet supersense, vs. the attribute supersense that has no consistent effect on modality shifts. for more plots, see appendix b. ## conclusion and discussion the goal of our paper was to investigate what semantic factors contribute to the difference in representational spaces of language-only models vs. multimodal models. our regression analysis confirmed previous findings that concreteness plays a major role in this difference @xcite . this is natural since imageability, the measurable manifestation of concreteness, is directly related to whether useful information about a concept can be inferred form visual data. however, other factors beyond abstractness contribute to the modality-based space contrasts as well. the most important factor here is taxonomical, as measured by the effect of wordnet lexicographer files. wordnet supersenses consistently affect semantic similarities in text-only models vs. l&v models: in particular, we found this for artifacts, quantities, possessions and communication lexical classes. lastly, sentiment-related lexical properties, most clearly valence, also affect the semantic similarity in language-only vs. multimodal spaces. recently, several studies in semantics and pragmatics have indicated interactions of connotational content with denotational meanings @xcite @xcite @xcite . our results can be interpreted as pointing in that direction too. still, the effect of sentiment is overall much smaller than the core denotational properties of the words in the lexical pair, as illustrated by the comparison of the combined vad to combined taxonomic features in we contribute to the understanding of different embedding spaces by demonstrating systematic differences between text-only vs. l&v models. many questions are however left for future research. for example, do the distinct properties of multimodal embeddings make them better suited for specific tasks, as @xcite argued for the relat-edness judgments of concrete nouns? in the light of kruszewski's finding @xcite that taxonomic information interacts strongly with referential compatibility between concepts, our findings on the role of taxonomic status on vector space structure suggests that the choice of multimodal vs. textual representations can be crucial for inference, especially for the difficult case of the neutral vs. contradiction distinction. finally, we note that the semantic factors we considered only explain a small part of the discrepancy between textual and l&v models. the rest must be attributed to other factors, such as random differences in the textual data used for model training as well as semantic phenomena outside the scope of our study. we hope that our study inspires further exploration of systematic differences between embedding models, both for visual grounding and beyond.