Investigating Word-Class Distributions in Word Vector Spaces (2020)

Abstract: This paper presents an investigation on the distribution of word vectors belonging to a certain word class in a pre-trained word vector space. To this end, we made several assumptions about the distribution, modeled the distribution accordingly, and validated each assumption by comparing the goodness of each model. Specifically, we considered two types of word classes – the semantic class of direct objects of a verb and the semantic class in a thesaurus – and tried to build models that properly estimate how likely it is that a word in the vector space is a member of a given word class. Our results on selectional preference and WordNet datasets show that the centroid-based model will fail to achieve good enough performance, the geometry of the distribution and the existence of subgroups will have limited impact, and also the negative instances need to be considered for adequate modeling of the distribution. We further investigated the relationship between the scores calculated by each model and the degree of membership and found that discriminative learning-based models are best in finding the boundaries of a class, while models based on the offset between positive and negative instances perform best in determining the degree of membership.

## introduction several studies have been successful in representing the meaning of a word with a vector in a continuous vector space (e.g., @xcite . these representations are useful for a range of natural language processing (nlp) tasks. the interpretation and geometry of the word embeddings have also attracted attention (e.g., @xcite . however, little attention has been paid to the distribution of words belonging to a certain word class in a word vector space, though empirical analysis of such a distribution provides a better understanding of word vector spaces and insight into algorithmic choices for several nlp tasks, including selectional preference acquisition and entity set expansion. figure 1 shows a 2d projection of word embeddings. we extracted 200 words that can be a direct object of the verb play (positive instances) and 1000 other words (negative instances) and projected their glove vectors @xcite into two dimensions using t-distributed stochastic neighbor embedding (t-sne) (van der @xcite . the plus symbols (+) represent the positive instances, and the squares ( ) represent the negative instances. this figure shows that the positive instances tend to be densely distributed around their centroid but they are not evenly distributed near the centroid in the 2d spaces. in this study, we aimed to understand how these positive instances are distributed in the pre-trained word vector spaces built by three representative generalpurpose models: cbow, skip-gram @xcite , and glove. more specifically, we attempted to determine the following: whether or not a simple centroid-based approach can provide a reasonably good model, whether or not considering the geometry of the distribution and the existence of subgroups is useful for modeling the distribution, and whether or not considering the negative instances is essential to achieve adequate modeling. to this end, we first tackled properly modeling the vector distribution to distinguish a possible member of a word class from others when a subset of the class members is given. note that although various approaches have been proposed to improve word vectors by taking knowledge related to word classes into account @xcite @xcite , we explored ways to model the distribution of word vectors rather than attempting to improve the word vectors themselves. we started with a centroid-based model, which is a simple but widely used way of representing a set of word vectors (e.g., @xcite ; woodsend and lapata 2015) and assumes that how likely a word in the vector space is a member of a word class is proportional to the proximity to the centroid vectors of the class members. we then explored models that take the geometry of the distribution and the existence of subgroups into account. here, we made two assumptions: vectors of words belonging to a certain word class are distributed with different variances depending on the direction, and most word sets will consist of several subgroups. we then explored the models that also consider negative instances. we assumed that the vectors of the words that do not belong to the target word class can be essential clues to distinguish a possible member of a word class from others. specifically, we explored a model based on the offset between positive and negative instances and discriminative learning-based models to investigate the impact of negative instances. furthermore, we investigated the relationship between the scores calculated by each model and the degree of membership using the @xcite dataset. the dataset contains typicality ratings for some instances of a category. through experiments, we found that discriminative learning-based models perform better at distinguishing a possible member of a word class from others, while the offset-based model achieves higher correlations with the degree of membership. ## related work the interpretation and geometry of word embeddings have attracted attention. @xcite reported that vector positions trained with skip-gram negative sampling (sgns) do not span the possible space uniformly but occupy a narrow cone instead. @xcite showed that constant vector offsets of word pairs can represent linguistic regularities. @xcite demonstrated that vector offsets can be used to derive a scalar relationship amongst adjectives. @xcite performed an analysis of subspaces in word embedding. these analyses suggest that a certain direction or subspace in the word vector space represents an aspect of the words and the possibility that a word class is distributed with different variances depending on the direction in the vector space. while we investigated ways to model the distribution of a set of words in pre-trained word vector spaces to validate several assumptions about the distribution, various approaches have been proposed to improve word embeddings by considering knowledge related to word classes into account. for example, @xcite proposed a method of refining vector representations using relational information from semantic lexicons by encouraging linked words to have similar vector representations. @xcite proposed an algorithm for improving the semantic quality of word vectors by injecting constraints extracted from lexical resources. glavaš and vulić (2018) use the linguistic constraints as training examples to learn an explicit specialization function with deep neural network architecture. there are also several studies that expand the method for acquiring a word vector to consider the uncertainty of a word meaning via gaussian models @xcite and word polysemy by introducing several vectors for each word @xcite @xcite . in this study, we only considered a vector for representing each word, but inspired by these studies, we explored models that can consider the geometry of the distribution and the existence of subgroups. the problem we tackled is similar to a selectional preference acquisition task. there have been a number of studies on selectional preference acquisition. @xcite presented an informationtheoretic approach that inferred selectional preferences based on the wordnet hypernym hierarchy. @xcite described a method that uses corpus-driven distributional similarity metrics for selectional preference induction. van de cruys (2014) investigated the use of neural networks for selectional preference acquisition. an entity set expansion task @xcite ) is also similar to our problem and has been well studied. for example, @xcite disambiguated entity word senses and alleviated semantic drift by extracting topic information from lda for entity set expansion. @xcite proposed a joint model for entity set expansion and attribute extraction. in this study, we seek to understand how these vectors are distributed in the pre-trained word vector space without using contextual or lexical information. a comparison with the state-of-the-art models for selectional preference induction and entity set expansion is beyond the scope of this work. ## problem formulation first, let us introduce the notation. w c is a subset of words that belong to the target word class c. w o is a subset of words that do not belong to the word class. w t is a target word that can be a member of the word class c but is not included in w c . v @xmath0. we normalize all the word vectors to unit length. foot_0 note that we select the words in w o to share the same grammatical category as the words in w c . our objective is to distinguish the word w t from the words in w o , given w c and v w . more specifically, we aim to find a scoring function f(w, w c ) that assigns a higher score to w t and lower scores to the words in w o . for example, suppose c is a class of words that can be a direct object of the verb play; w c , w o and w t will be as follows: w @xmath1, part, game, golf, tennis}, w @xmath2, apple, milk, arch, idea}, and w @xmath3. our objective is to find a scoring function that assigns a higher score to basketball than to school, apple, milk, arch, and idea. ## models we will start with a centroid-based model (cent) that measures the score between a word w and a word set w c by calculating the cosine similarity between the word vector and the centroid vector of the word vectors in the word set (figure 2-(a) ). the scoring function can be written as: cent provides a reasonable baseline, but it does not take the geometry of the distribution of the word vectors into account. therefore, we introduce a simple gaussian model (gm) to represent the distribution of word vectors belonging to a word class c (figure 2-(b) ). the scoring function is as follows: next, we introduce a gaussian mixture model (gmm) to take the existence of subgroups in a word class c into account (figure 2-(c )). the scoring function can be written as: as the last model without negative instances, we adopt a one-class support vector machine (svm) @xcite )-based model (1-svm) to clarify the importance of the negative instances. we select the kernel from {linear, cubic polynomial, rbf} and tune the parameter @xmath4.05, 0.10, . . . , 0.50} by performing cross-validation. note that models without negative instances learn a decision function for outlier detection: classifying new data as similar or different to the given positive instances. next, we explore models that also leverage negative instances. here, we introduce a word set w n as negative instances, where w n consists of words that are not included in either w c or w o . we select the words in w n to share the same grammatical category as the words in w c as well as w o . both w o and w n consist of words that are not included in w c , but their roles are different. while words in w o are used as negative instances in the estimation, words in w n are used as negative instances for modeling the word-class distribution. as the first model with negative instances, we introduce a model based on the offset between positive and negative instances (offset). this model is inspired by the kim and de marneffe (2013)'s work, which demonstrates that vector offsets can be used to derive adjectival scales. we assume that the vector offset between the centroid of the positive instances and that of the negative instances represents the degree of membership in the vector space (figure 2-(d) ). the scoring function of offset is as follows: now let us move on to discriminative learningbased models. in this study, we chose a support vector machine with a linear kernel (svm l ) or a radial basis function (rbf) kernel (svm r ). we only used word vectors as the input of these models and regard the decision function as the scoring function. we tuned the parameter @xmath5.1, 0.2, 0.5, 1, 2, 5, 10} and class weight for positive instances @xmath6, 2, 4, 8} for svm l and the parameter @xmath7.2, 0.5, 1, 2, 5}, γ ∈ {0.2, 0.5, 1, 2}, and class weight for positive instances @xmath8, 2, 4, 8} for svm r by performing cross-validation on w c and w n . note that we wanted to determine the usefulness of negative instances in modeling the distribution of word vectors; thus we make no assertions that these are optimal models. ## conclusion and future work we investigated the distribution of words that belong to a certain word class in a pre-trained generalpurpose word vector space. the experimental results show that a centroid-based approach cannot provide a reasonably good model and considering the geometry of the distribution and the existence of subgroups is useful for modeling the distribution in some cases. however, the impact is limited, and the negative instances must be taken into account for adequate modeling. the results indicate that just observing the distribution of positive instances is not enough to understand the geometry of word embedding spaces. furthermore, we investigated the relationship between the score calculated by each model and the degree of membership and demonstrated that, while discriminative learning-based models can distinguish a possible member of a word class from others, the offsetbased model achieves higher correlations with the degree of membership. the investigation in this study leveraged only general-purpose word vectors to represent the meaning of a word. however, several studies have expanded the method for acquiring a word vector to account for the uncertainty of word meanings and word polysemy (e.g., @xcite ). in addition, contextualized word embeddings have been shown to be very effective on a range of nlp tasks @xcite . furthermore, @xcite reported that word embeddings learned in several tasks are biased towards word frequency: the embeddings of high-frequency and low-frequency words lie in different subregions of the embedding space. thus, in the future, we will take the uncertainty, polysemy, and context sensitivity of the word meanings and the frequency of words into account and explore better ways of modeling the word-class distributions in semantic vector spaces.