comp-syn: Perceptually Grounded Word Embeddings with Color (2020)

Abstract: Popular approaches to natural language processing create word embeddings based on textual co-occurrence patterns, but often ignore embodied, sensory aspects of language. Here, we introduce the Python package comp-syn, which provides grounded word embeddings based on the perceptually uniform color distributions of Google Image search results. We demonstrate that comp-syn significantly enriches models of distributional semantics. In particular, we show that(1) comp-syn predicts human judgments of word concreteness with greater accuracy and in a more interpretable fashion than word2vec using low-dimensional word–color embeddings ,and (2) comp-syn performs comparably to word2vec on a metaphorical vs. literal word-pair classification task. comp-syn is open-source on PyPi and is compatible with mainstream machine-learning Python packages. Our package release includes word–color embeddings forover 40,000 English words, each associated with crowd-sourced word concreteness judgments.

## introduction the embodied cognition paradigm seeks to ground semantic processing in bodily, affective, and social experiences. this paradigm explains how sensory information contributes to semantic processing, either through metaphors involving references to sensory experience @xcite or through the simulation of sensory experience in mental imagery @xcite . for example, color is pervasive in linguistic metaphors (e.g., "her bank accounts are in the red"), and primes a range of affective and interpretative responses @xcite . extant methods in computational linguistics are limited in their ability to advance the embodied cognition paradigm due to their focus on text, which often precludes multi-modal analyses of images and other forms of sensory data-including color-involved in human meaning-making activities. distributional semantics is a particularly prominent approach, wherein textual co-occurrence patterns are used to embed words in a high-dimensional space; the resulting "distance" between word embeddings correlates with semantic similarity @xcite . although leveraging neural networks @xcite 2013) and syntactic information @xcite has led to recent progress, popular models like word2vec lack firm grounding in their use of sensory information. moreover, high-dimensional word embeddings created by neural networks are difficult to interpret @xcite and require long training periods @xcite . thus, it remains difficult to reconcile models of distributional semantics with theories of embodied semantics. to address these limitations, multi-modal approaches to distributional semantics combine text and image data. however, these models continue to infer semantic associations using high-dimensional wordplus-image embeddings; their interpretability therefore remains an issue @xcite @xcite . with respect to color, multi-modal models operate in standard colorspaces like rgb that are not perceptually uniform @xcite and embed color in a manner that combines it with complex, multidimensional spatial information. here, we introduce a novel word embedding method based on color that is explicitly interpretable with respect to theories of embodied cognition. we build on work which shows that color distributions in online images reflect both affective and semantic similarities among words in abstract domains (e.g., in the domain of academic disciplines), while also characterizing human judgments of concept concreteness @xcite . we present the python package comp-syn foot_0 , which allows users to explore word-color embeddings based on the perceptually uniform color distributions of google image search results. we provide embeddings for a set of 40,000 common english words, and we benchmark the performance of our model using crowd-sourced human concreteness ratings @xcite . we show that comp-syn complements the performance of text-based distributional semantics models by providing an interpretable embedding that both (1) predicts human judgments of concept concreteness, and (2) distinguishes metaphorical and literal word pairs. we use google rather than a curated image database such as imagenet @xcite because popular datasets are heavily biased toward concrete objects, which limits their applicability in abstract semantic domains. moreover, google image search results reflect content that users interact with most frequently @xcite , underscoring their relevance for connecting distributional properties of words and images to human semantic processing. to ensure that our approach is entirely unsupervised, we do not select particular features of images when measuring their color distributions. this approach avoids importing pre-determined semantic notions into our analysis and connects our method to cognitive theories that attribute aesthetic relevance to recognizable features in both the background and foreground of images @xcite @xcite . ## results ## conclusion and future work we have presented comp-syn, a new python package that provides perceptually grounded color-based word embeddings. these embeddings are interpretable with respect to theories of embodied cognition because (1) comp-syn represents color in a fashion that emulates human perception, and (2) comp-syn leverages google image search results that human users dynamically interact with and produce. by linking comp-syn with human concreteness judgments for 40,000 common english words, our package provides a multi-modal playground for exploring grounded semantics. we demonstrated that comp-syn enriches popular distributional semantics models in both word concreteness prediction and metaphorical word-pair classification. a myriad of comp-syn applications await, including colorbased classification of text genres and the characterization of sensory imagery in everyday language.