Seeing the advantage: visually grounding word embeddings to better capture human semantic knowledge (2022)

Abstract: Distributional semantic models capture word-level meaning that is useful in many natural language processing tasks and have even been shown to capture cognitive aspects of word meaning. The majority of these models are purely text based, even though the human sensory experience is much richer. In this paper we create visually grounded word embeddings by combining English text and images and compare them to popular text-based methods, to see if visual information allows our model to better capture cognitive aspects of word meaning. Our analysis shows that visually grounded embedding similarities are more predictive of the human reaction times in a large priming experiment than the purely text-based embeddings. The visually grounded embeddings also correlate well with human word similarity ratings. Importantly, in both experiments we show that the grounded embeddings account for a unique portion of explained variance, even when we include text-based embeddings trained on huge corpora. This shows that visual grounding allows our model to capture information that cannot be extracted using text as the only source of information.

## introduction distributional semantic models create word representations that quantify word meaning based on the idea that a word's meaning depends on the contexts in which the word appears. such representations (also called embeddings) are widely used as the linguistic input for computational linguistic models, with research showing that they can account for response times in lexical decision tasks @xcite @xcite , decode brain data @xcite , account for brain activity during text comprehension @xcite , and correlate with human judgements of word similarity @xcite @xcite . while such embeddings have proven useful, they are not cognitively plausible as creating high quality embeddings requires billions of word tokens. for instance, the glove embeddings developed by @xcite are trained on 840 billion words. it would require a human 80 years of constant reading at about 330 words per second to digest that much information. obviously, humans are able to understand language after much less exposure, and furthermore, their sensory experience is much richer than solely reading texts. embodied cognition theory poses that our conceptual knowledge is based on the entirety of our sensory experience @xcite . for instance, reading the word dog elicits sensory experiences we have with dogs, such as their sound and how they look. embodied cognition theory thus assumes that all our sensory experiences contribute to our conceptual knowledge and processing, which should be reflected in human behaviour. early priming studies have indeed found that visual similarities can elicit priming effects @xcite . if visual features are part of our conceptual knowledge, word embeddings incorporating visual features should be able to explain human behavioural data to a degree unattainable by purely text-based methods (that is, if we assume visual sensory experiences can never be fully captured by textual descriptions). that is why recent research has taken an interest in multimodal word embeddings, combining text with a second source of information, resulting in visually grounded embeddings (vges) in the case of visual information. ## methods in our experiments, we compare the vges from our own model with three well known text-based distributional semantic models: fasttext @xcite , word2vec @xcite and glove @xcite . for the purpose of this study, we take two approaches: 1) we train our own text-based distributional models to allow for a fair comparison to the vges, and 2) we use the pretrained models to investigate whether our vges capture semantic information that even models trained on large text corpora do not. the code used in this study can be found at https://github.com/dannymerkx/ speech2image/tree/cmcl2022 2.1 training data mscoco is a database intended for training image recognition, segmentation and captioning models @xcite . it has 123,287 images and 605,495 written english captions, that is, five captions paired to each image. captions were collected by asking annotators to describe what they saw in the picture. five thousand images (25,000 captions) are reserved as a development set. the captions are provided in tokenised format. in order to use them in our models we only decapitalised all words and removed the punctuation at the end of each sentence. this results in a total of 6,184,656 word tokens and 28,415 unique word types, to which we add start-and end-of-sentence tokens for training our visually grounded model. the images are pre-processed by resizing the images such that the shortest side is 256 pixels, while keeping the original aspect ratio. we take ten 224 by 224 crops of the image: one from each corner, one from the middle and the same five crops for the mirrored image. we use resnet-152 @xcite pretrained on imagenet to extract visual features from these ten crops and then average the features of the ten crops into a single vector with 2,048 features. these features are extracted by removing resnet's classification layer and taking the activations of the penultimate layer. ## discussion we created visually grounded embeddings using a caption-image retrieval model in order to test if these embeddings can capture information about word meaning that text-based approaches cannot. importantly, by testing our vges on human behavioural measures typically thought to rely on conceptual/semantic knowledge, we test a central idea of embodied cognition theory, namely that our visual experiences contribute to our conceptual knowledge. ## conclusion we set out to test an end-to-end approach to combining visual and textual input in a single embedding, trained on a cognitively plausible amount of data. the results from our two experiments suggest that vges capture aspects of word meaning that text-based approaches cannot. even though we include word embeddings trained on corpora several orders of magnitude greater than any human's exposure to language, our vges still explain a unique portion of variance in both human behavioural measures. while our results indicate that visual grounding can provide complementary information for certain words, it may not play a role in our conceptual knowledge of rare, abstract words, as shown by our results on the rarewords corpus. similar to @xcite this then does not support the strongest formulations of embodied cognition theory which suggest total equivalence between conceptual and sensorimotor processing @xcite . of course, one could always claim that it is just current word-embedding models that do not fully capture word meaning yet. however, given that vges trained on a relatively small amount of visual data can complement text-based embeddings, we do not think even larger text-corpora or more complex embedding models can ever fully capture human semantic knowledge. the human experience is rich and varied, and our computational models can never fully capture human word knowledge while ignoring visual aspects of this experience.