A Mixture-of-Experts Model for Learning Multi-Facet Entity Embeddings (2020)

Abstract: Various methods have already been proposed for learning entity embeddings from text descriptions. Such embeddings are commonly used for inferring properties of entities, for recommendation and entity-oriented search, and for injecting background knowledge into neural architectures, among others. Entity embeddings essentially serve as a compact encoding of a similarity relation, but similarity is an inherently multi-faceted notion. By representing entities as single vectors, existing methods leave it to downstream applications to identify these different facets, and to select the most relevant ones. In this paper, we propose a model that instead learns several vectors for each entity, each of which intuitively captures a different aspect of the considered domain. We use a mixture-of-experts formulation to jointly learn these facet-specific embeddings. The individual entity embeddings are learned using a variant of the GloVe model, which has the advantage that we can easily identify which properties are modelled well in which of the learned embeddings. This is exploited by an associated gating network, which uses pre-trained word vectors to encourage the properties that are modelled by a given embedding to be semantically coherent, i.e. to encourage each of the individual embeddings to capture a meaningful facet.

## introduction entity embeddings are vector space representations of the entities from a given domain. such representations are commonly used in cognitive science, where they are referred to as semantic spaces or conceptual spaces @xcite . as another example, the field of information retrieval also has a long tradition of using vector space representations @xcite . in the field of natural language processing (nlp), recent years have witnessed an explosion of applications that rely on entity embeddings. for instance, entity embeddings are now commonly used for injecting background knowledge @xcite , and as core representations for recommender systems @xcite and entity-focused search @xcite @xcite . entity embeddings are learned using a variety of different inputs, ranging from human similarity judgements, to text descriptions, web tables and images. regardless of how they are learned, entity embeddings can essentially be viewed as compact encodings of a similarity relation. indeed, while many embeddings exhibit various interesting linear regularities, such regularities are the result of the structure of the similarity relation that is used for learning the embedding @xcite . similarity is inherently multi-faceted, with the importance of different facets being context dependent. for instance, two movies can be similar because they belong to the same genre or because they are about the same historic event, among many others. however, these different facets of similarity are not reflected in the structure of standard entity embeddings. to see why this is sub-optimal, consider the problem of concept induction: given a small set of entities e 1 , ..., e k , identify other entities that are of the same kind. for instance, given the examples barcelona, madrid, alicante, valid completions would be other spanish cities. the problem of concept induction underpins many of the applications in which entity embeddings are used, including knowledge base completion and recommendation. in cases where this work is licensed under a creative commons attribution 4.0 international license. license details: http: //creativecommons.org/licenses/by/4.0/ . the given set of entities is small, the result will strongly depend on the similarity relation encoded by the given entity embedding: given a few entities, we can do little else than selecting the nearest neighbors of their (averaged) entity vectors. however, if we have several embeddings of the considered entities, each capturing different facets, then we can solve the concept induction task by first identifying the most relevant facet(s), and thus rely on a form of similarity that is relevant for the given concept. the problem of learning separate facet-specific embeddings is related to disentangled representation learning. while this latter problem has already received considerable attention, most existing work has focused on (semi)-supervised settings, primarily in the visual domain. unsupervised approaches for disentangled representation learning generally need strong inductive biases @xcite . in the text domain, most work has focused on separating style or sentiment from content. however, one recent exception is @xcite , where an iterative method is proposed to decompose a given entity embedding into facet-specific vector spaces. to provide the required inductive bias, they first determine which properties are captured by the given entity embedding. these properties correspond to words from text descriptions of the entities, whose occurrence can be predicted from the entity vectors. to identify words that are likely to describe properties from the same facet, they rely on the intuition that such properties should have similar word vectors, in a given pre-trained word embedding. the experimental results from @xcite show that learning facet-specific embeddings is indeed helpful for concept induction. however, their method is applied to entity embeddings that have been learned from text descriptions using multi-dimensional scaling (mds), which has two important limitations. first, mds has a quadratic space complexity, which makes it unsuitable for large domains. second, and most fundamentally, they crucially rely on the assumption that facets of interest correspond to linear sub-spaces of the initial entity embedding. as another limitation of their method, the assumption that facets can be identified with clusters in a word embedding space seems too strong. while words that describe properties of the same kind (e.g. different names of movie genres) are indeed often clustered together in a word embedding, the range of words that are relevant to a given facet is usually more varied (e.g. adjectives such as scary are relevant when modelling genre, but this word may not be clustered together with genre names). to address these issues, we propose a method that directly learns multifacet entity embeddings from text descriptions. to this end, we use a mixture-of-experts formulation @xcite , in which the experts essentially correspond to glove models @xcite , each focusing on a subset of the vocabulary. the decision on which words are modelled by which experts is made by a so-called gating network, which uses pre-trained word vectors as input. in this way, we can capture the intuition that words which are relevant to the same facet typically have similar word embedding representations, without having to assume that all such word appear in a single cluster. the problem of learning conceptual spaces from data has only received limited attention. inspired by conceptual spaces, @xcite introduced a method for structuring a given entity embedding using interpretable (but non-orthogonal) dimensions. this method was used in the approach by @xcite , which consists of (i) identifying interpretable dimensions in the given entity embedding, (ii) clustering the words describing these dimensions, (iii) identifying the linear subspace that best corresponds to the most dominant cluster, (iv) repeating the same method on the orthogonal complement of this subspace. somewhat related, @xcite propose a supervised method to decompose word embeddings into subspaces that capture particular aspects of word meaning, such as sentiment polarity or part-of-speech. the idea of decomposing a word embedding space into sub-spaces is also central to the method from @xcite , which is aimed at distinguishing synonyms from antonyms. within a broader context, @xcite propose a method to group numerical features into domains, with the aim of generating better linguistic descriptions of numerical data. ## model description the main idea underpinning the mixtures-of-experts (moe) model @xcite is to train a neural network by (i) learning a soft partition of the feature space and (ii) training a separate neural network for each partition class. the individual neural networks, referred to as experts, are thus specialized towards the examples from the corresponding partition class. these experts are jointly trained with a so-called gating network, which is used to determine the (soft) partition. to apply this model to our setting, we thus need to determine the structure of the gating network and the nature of the experts. our aim is to learn facet-specific entity embeddings from the bag-of-words representations (bow) of a given set of entities. to apply the moe model to this problem setting, we need an embedding method that can be formulated as a classification or regression problem. moreover, to allow for an effective gating network, we need the ability to efficiently determine how well different properties are captured by the different entity embeddings. to address both issues, we build on the glove word embedding model @xcite , which is a common choice for learning entity embeddings from bow representations @xcite . using the notations and terminology of entity embeddings, the glove model can be formulated as follows: here e i represents the embedding of entity e i , wj is a representation of the word w j , b i and bj are bias terms, x ij is the number of occurrences of w j in the bow representation of e i , and the weight f (x ij ) is aimed at reducing the impact of rare words. the term g j captures how well the entity embedding is modelling the word w j . similar to @xcite , we found that words which are modelled well, i.e. for which the loss term g j is low, tend to correspond to semantically meaningful properties. the main idea of our method is to learn multiple glove embeddings (i.e. experts), where each embedding will be specialized towards a subset of all words. the key challenge is to train these embeddings such that the properties captured by a given embedding form a semantically meaningful facet or domain. for example, when learning a representation of movies, we would expect to see one glove expert that focuses on genre (e.g. capturing words such as horror, zombie or funny). what makes this problem particularly challenging is that properties from different facets are often correlated (e.g. particular actors may be strongly associated with a particular movie genre). this is in accordance with the theory of conceptual spaces, but it means that a strong inductive bias is needed to learn these representations. following alshaikh et al. (2019) , we rely on pre-trained word vectors to provide this bias. in particular, we rely on the assumption that whenever a word is related to a given facet, words with similar embeddings tend to be related as well. this assumption is less strong than the assumption from @xcite , where each facet was assumed to correspond to a single cluster. ## experiments we experimentally analyze the performance of the proposed mixture-of-experts (moe) model. our main focus is on showing that learning facet-specific embeddings is useful compared to learning standard embeddings. we also compare our method with the approach from @xcite . datasets. we use the movies and place types datasets from @xcite and the buildings dataset from @xcite . these datasets respectively contain bow descriptions of movies (obtained from reviews), place types (obtained from flickr tags) and buildings (obtained from wikipedia articles). each of these datasets is associated with a number of classification problems, which are listed in second, we compiled a new dataset from the english wikipedia. in particular, we selected the 100 000 wikipedia concepts with the longest articles, which approximately corresponds to those concepts whose wikipedia article contains more than 200 words, after removing stop words and words that appear less than 10 times in the collection. as classification tasks for the wikipedia dataset, we first consider the problem of predicting the wikidata semantic type of the wikipedia entities. in particular, we identified 13 semantic types that occur sufficiently frequently, each having at least 2000 instances in our collection. in addition to these semantic types, we extracted nine attributes from wikidata, for which a value was specified for a sufficient number of instances: three attributes for movie entities and two attributes for each of music, business and human. the considered classification problems are listed in ## conclusion this paper has introduced a method for jointly learning a number of facet-specific low-dimensional entity embeddings. to the best of our knowledge, this is the first approach for learning such representations that is both scalable and unsupervised. we have presented experimental results which show that learning facet-specific spaces can be highly beneficial. while we have focused on bag-of-words input representa-tions in this paper, in future work it would be interesting to see how similar strategies could be applied to document embedding strategies based on bert @xcite , or related language models.