On the Geometry of Concreteness (2022)

Abstract: In this paper we investigate how concreteness and abstractness are represented in word embedding spaces. We use data for English and German, and show that concreteness and abstractness can be determined independently and turn out to be completely opposite directions in the embedding space. Various methods can be used to determine the direction of concreteness, always resulting in roughly the same vector. Though concreteness is a central aspect of the meaning of words and can be detected clearly in embedding spaces, it seems not as easy to subtract or add concreteness to words to obtain other words or word senses like e.g. can be done with a semantic property like gender.

## introduction in the current paper we aim to shed some light on the way concreteness and abstractness are represented in word embeddings. this might help to better understand the concept of concreteness that seems to be an important semantic property used to explain various phenomena in language and language understanding. ultimately, it might also contribute a little bit to the understanding of the semantic spaces in which we embed words for many tasks. ## related work word embeddings are widely used as a proxy for the meaning of words but in fact word embeddings are chiefly compact representations of the contexts in which they occur. since concrete words occur preferably in the context of other concrete words and since concrete words are used as object to sensory verbs we expect that concreteness can be found in word embeddings. indeed a number studies have shown the presence of concreteness in word embeddings: @xcite try to find low-dimensional feature representations of words in which at least some dimensions correspond to interpretable properties of words. one of these dimensions is concreteness. for training and testing they use google news embeddings and two subsets of frequent words from the norms of @xcite . for their test set of 8,694 frequent words they found a moderate correlation with the human judgments (kendall's τ = 0.623). similarly, hollis and westbury (2016) investigated which dimensions of word embeddings correlate to one of the classical word norms. they found no direct correlations, but after reducing the number of dimensions for a set of words by applying singular value decomposition, they found a strong correlation between one of the dimensions and concreteness. @xcite train regression models on word embeddings to predict concreteness values, thus showing that concreteness information is present in the embeddings. ## materials the answers to the research questions might depend on the embeddings we use. nevertheless, we will restrict the experiments to just two embeddings, one for english and one for german, and for the moment being assume that results for other embeddings will be similar. for english we use the 300 dimensional fasttext embeddings without subword information trained on the common crawl with 600 billion tokens. for german we also use 300 dimensional fasttext embeddings trained on the common crawl and wikipedia. both embeddings are available at the fasttext site ( https://fasttext.cc/ ). the concreteness values are taken from @xcite for english and from the merged dataset from charbonnier and wartena (2020) for german. since concreteness is most clearly defined for nouns, from both datasets we use only nouns for which we also have embeddings. in the data from @xcite the words are rated between 1.0 and 5.0. the ratings for the german data range from 1.0 to 7.0. as examples of concrete nouns we take for the english data all nouns rated above 4.0 and for german all nouns rated above 6.0. as clearly abstract nouns we use nouns rated below 2.7 for english and rated below 4 for german. this results in the numbers given in in this section we will compare different methods to build prototypical vectors for concreteness and abstractness. ## diversity of concrete and abstract words as discussed above it has been observed that abstract words occur in more diverse contexts than concrete words. does this also mean that abstract words are more diverse? i.e., can words be concrete just in one way but abstract in many different ways? to answer this question we selected randomly 100 words from our set of concrete and 100 from the set of abstract words. we compute the average cosine similarity for all pairs of words within each set and within the union of both sets. the results are given in ## concreteness and regular polysemy a word like school can refer to the schoolhouse or to the educational institution. in our sets of word embeddings there is only one embedding for all meanings of the word school, even including the sense of a group (as in a school of fish), a group of artists or thinkers and even the verb to school. we would hope that if we add a little bit of concreteness to the embedding of school, we get an embedding that is a bit closer to the embedding of schoolhouse and if we add some abstractness, the embedding becomes more similar to other abstract concepts from education. as a first indication to see whether this is indeed the case, we visualize the distances between a few ambiguous words (school, university and hospital for english and schule (school), universität (university) and fabrik (factory) for german) along with some related concrete and abstract words. for each ambiguous word w we use the original embedding v w as well as v w + 0.2v ca and v w -0.2v ca . we add 0.2 vca since 0.2 is roughly the length of the projection of the most abstract and the most concrete words on vca . in the visualization the variants are labeled with the original word and either an a or c. the projection in a two-dimensional space is done with tsne (van der @xcite . the results are shown in figure 1 1 . since we added clearly abstract and concrete words concreteness becomes a clear dimension in the visualization. for all words we see that the concrete variants indeed moved into the direction of the related concrete words and similar for the abstract variants. in order to know whether the concrete and abstract variants of the embeddings really become more similar to synonyms of the respective senses we selected 10 ambiguous words for english and german along with a closely related word for the abstract and for the concrete sense. for german we selected 10 words that are ambiguous between a building (or location) and an institution. for english we selected words that either denote a process or an actor involved in that process. here we tried to select words that do not have too many senses, are predominantly used as noun and for the related words we tried to find synonyms that do not have the same ambiguity. most of the related words were taken from the synsets in wordnet @xcite to make the choices somewhat more objective. now for each word we add (subtract) 0.2 vca and determine how much the resulting vector is closer to the embedding of the related concrete (abstract) word than the original vector. the results for are give in tables 6 and 7 . in all cases we see that the improvement is very small or even negative. for some of the word pairs, like parlament -parlamentsgebäude (parliament -parliament's house) or schule -schulgebäude, it seems that the second word is a real synonym of the building sense of the first word and we would expect that the cosine similarity would be much larger when adding the concreteness to the general vector. thus we have to conclude that though the senses of these ambiguous german words clearly have different degrees of concreteness, the difference between the senses is much more than just the concreteness. for the english words that are ambiguous between a process and an entity, adding concreteness in two cases even makes the pairs more dissimilar and adding abstractness only in one case makes the word more similar to a synonym of the process reading. ## conclusion we have seen that concreteness can be identified as a direction in the word embedding space. various methods, based on many words with concreteness values or just on a view highly concrete words give almost the same vector for concreteness. moreover, the cosine of these vectors with the embeddings of words correlates strongly with the concreteness judgments of human subjects of these words. thus our first research question can be answered positively. furthermore, we see that concreteness and abstractness are quite symmetric properties. we can