Word Embeddings as Tuples of Feature Probabilities (2020)

Abstract: In this paper, we provide an alternate perspective on word representations, by reinterpreting the dimensions of the vector space of a word embedding as a collection of features. In this reinterpretation, every component of the word vector is normalized against all the word vectors in the vocabulary. This idea now allows us to view each vector as an n -tuple (akin to a fuzzy set), where n is the dimensionality of the word representation and each element represents the probability of the word possessing a feature. Indeed, this representation enables the use fuzzy set theoretic operations, such as union, intersection and difference. Unlike previous attempts, we show that this representation of words provides a notion of similarity which is inherently asymmetric and hence closer to human similarity judgements. We compare the performance of this representation with various benchmarks, and explore some of the unique properties including function word detection, detection of polysemous words, and some insight into the interpretability provided by set theoretic operations.

## introduction word embedding is one of the most crucial facets of natural language processing (nlp) research. most non-contextualized word representations aim to provide a distributional view of lexical semantics, known popularly by the adage "a word is known by the company it keeps" @xcite . popular implementations of word embeddings such as word2vec @xcite and glove @xcite aim to represent words as embeddings in a vector space. these embeddings are trained to be oriented such that vectors with higher similarities have higher dot products when normalized. some of the most common methods of intrinsic evaluation of word embeddings include similarity, analogy and compositionality. while similarity is computed using the notion of dot product, analogy and compositionality use vector addition. however, distributional representations of words over vector spaces have an inherent lack of interpretablity @xcite . furthermore, due to the symmetric nature of the vector space operations for similarity and analogy, which are far from human similarity judgements @xcite . other word representations tried to provide asymmetric notions of similarity in a noncontextualized setting, including gaussian embeddings @xcite and word similarity by dependency @xcite . however, these models could not account for the inherent compositionality of word embeddings @xcite . moreover, while work has been done on providing entailment for vector space models by entirely reinterpreting word2vec as an entailment based semantic model @xcite , it requires an external notion of compositionality. finally, word2vec and glove, as such, are meaning conflation deficient, meaning that a single word with all its possible meanings is represented by a single vector @xcite . sense representation models in noncontextualized representations such as multi-sense skip gram, by performing joint clustering for local word neighbourhood. however, these sense representations are conditioned on non-disambiguated senses in the context and require additional conditioning on the intended senses @xcite . in this paper, we aim to answer the question: can a single word representation mechanism account for lexical similarity and analogy, compositionality, lexical entailment and be used to detect and resolve polysemy? we find that by performing column-wise normalization of word vectors trained using the word2vec skip-gram negative sampling regime, we can indeed represent all the above characteristics in a single representation. we interpret a column wise normalized word representation. we now treat these representations as fuzzy sets and can therefore use fuzzy set theoretic operations such as union, intersection, difference, etc. while also being able to succinctly use asymmetric notions of similarity such as k-l divergence and cross entropy. finally, we show that this representation can highlight syntactic features such as function words, use their properties to detect polysemy, and resolve it qualitatively using the inherent compositionality of this representation. in order to make these experiments and their results observable in general, we have provided the code which can be used to run these operations. the code can be found at https://github. com/alokdebnath/fuzzy_embeddings . the code also has a working command line interface where users can perform qualitative assessments on the set theoretic operations, similarity, analogy and compositionality which are discussed in the paper. ## related work the representation of words using logical paradigms such as fuzzy logic, tensorial representations and other probabilistic approaches have been attempted before. in this section, we uncover some of these representations in detail. @xcite introduced measures of distributional similarity to improve the probability estimation for unseen occurrences. the measure of similarity of distributional word clusters was based on multiple measures including euclidian distance, cosine distance, jaccard's coefficient, and asymmetric measures like α-skew divergence. bergmair (2011) used a fuzzy set theoretic view of features associated with word representations. while these features were not adopted from the vector space directly, it presents a unique perspective of entailment chains for reasoning tasks. their analysis of inference using fuzzy representations provides interpretability in reasoning tasks. @xcite presents a tenosrial calculus for word embeddings, which is based on compositional operators which uses vector representation of words to create a compositional distributional model of meaning. by providing a categorytheoretic framework, the model creates an inherently compositional structure based on distributional word representations. however, they showed that in this framework, quantifiers could not be expressed. @xcite refers to a notion of general formal semantics inferred from a distributional representation by creating relevant ontology based on the existing distribution. this mapping is therefore from a standard distributional model to a set-theoretic model, where dimensions are predicates and weights are generalised quantifiers. @xcite developed functional distributional semantics, which is a probabilistic framework based on model theory. the framework relies on differentiating and learning entities and predicates and their relations, on which bayesian inference is performed. this representation is inherently compositional, context dependent representation. ## background: fuzzy sets and fuzzy logic in this section, we provide a basic background of fuzzy sets including some fuzzy set operations, reinterpreting sets as tuples in a universe of finite elements and showing some set operations. we also cover the computation of fuzzy entropy as a bernoulli random variable. a fuzzy set is defined as a set with probabilistic set membership. therefore, a fuzzy set is denoted as @xmath0, µ a (x)), @xmath1, where x is an element of set a with a probability µ a (x) such that 0 ≤ µ @xmath2, and ω is the universal set. if our universe ω is finite and of cardinality n, our notion of probabilistic set membership is constrained to a maximum n values. therefore, each fuzzy set a can be represented as an n-tuple, with each member of the tuple a[i] being the probability of the ith member of ω. we can rewrite a fuzzy set as an n-tuple @xmath3, ∀@xmath4, such that |a | = |ω|. in this representation, a[i] is the probability of the ith member of the tuple a. we define some common set operations in terms of this representation as follows. the notion of entropy in fuzzy sets is an extrapolation of shannon entropy from a single variable on the entire set. formally, the fuzzy entropy of a set s is a measure of the uncertainty of the elements belonging to the set. the possibility of a member x belonging to the set s is a random variable x s i which is true with probability (p s i ) and f alse with probability (1 -p s i ). therefore, x s i is a bernoulli random variable. in order to compute the entropy of a fuzzy set, we sum the entropy values of each x s i : this formulation will be useful in section 4.4 where we discuss two asymmetric measures of similarity, cross-entropy and k-l divergence, which can be seen as a natural extension of this formulation of fuzzy entropy. ## representation and operations in this section, we use the mathematical formulation above to reinterpret word embeddings. we first show how these word representations are created, then detail the interpretation of each of the set operations with some examples. we also look into some measures of similarity and their formulation in this framework. all examples in this section have been taken using the google news negative 300 vectors foot_0 . we used these gold standard vectors ## experiments and results in this section, we present our experiments and their results in various domains including similarity, analogy, function word detection, polysemy detection, lexical entailment and compositionality. all the experiments have been conducted on established datasets. ## conclusion in this paper, we presented a reinterpretation of distributional semantics. we performed a columnwise normalization on word vectors, such that each value in this normalized representation represented the probability of the word possessing a feature that corresponded to each dimension. this provides us a representation of each word as a tuple of feature probabilities. we find that this representation can be seen as a fuzzy set, with each probability being the function of the projection of the original word vector on a dimension. considering word vectors as fuzzy sets allows us access to set operations such as union, intersection and difference. in our modification, these operations provide the product, disjoint sum and difference of the word representations, feature wise. using qualitative examples, we show that our representation naturally captures an asymmetric notion of similarity using feature difference, from which known asymmetric measures can be easily constructed, such as cross entropy and k-l divergence. we qualitatively show how our model accounts for polysemy, while showing quantitative proofs of our representation's performance at lower dimensions in similarity, analogy, compositionality and function word detection. we hypothesize that lower dimensions are more suited for our representation as sparsity increases with higher dimensions, so the significance of feature probabilities reduces. this sparsity causes a diffusion of the probabilities across multiple features. through this work, we aim to provide some insights into interpreting word representations by showing one possible perspective and explanation of the lengths and projections of word embeddings in the vector space. these feature representations can be adapted for basic neural models, allowing the use of feature based representations at lower dimensions for downstream tasks.