Polar Embedding (2021)

Abstract: Hierarchical relationships are invaluable information for many natural language processing (NLP) tasks. Distributional representation has become a fundamental approach for encoding word relationships, particularly embeddings in hyperbolic space showed great performance in representing hierarchies by taking advantage of their spatial properties. However, most machine learning systems do not suppose to use in such complex non-Euclidean geometries. To achieve hierarchy representations in commonly used Euclidean space, we propose Polar Embedding that learns word embeddings with the polar coordinate system. Utilizing characteristics of polar coordinates, the hierarchy of words is expressed with two independent variables: radius (generality) and angles (similarity), and their variables are optimized separately. Polar embedding shows word hierarchies explicitly and allows us to use beneficial resources such as word frequencies or word generality annotations for computing radiuses. We introduce an optimization method for learning angles in limited ranges of polar coordinates, which combining a loss function controlling gradient and distribution uniformization. Experimental results on hypernymy datasets indicate that our approach outperforms other embeddings in low-dimensional Euclidean space and competitively performs even with hyperbolic embeddings, which possess a geometric advantage.

## introduction a hierarchy is structured information that enables us to understand a specific object in a general sense (e.g., dog is one instance of mammal). such generalization capability is or will be a basis of intelligent systems such as comprehending causality @xcite , common sense @xcite , and logic @xcite . for example, species information (e.g., carnivora vs. herbivore) will be useful when predicting behavior of animals. another example is when developing a question answering system. hierarchical relations of words enable the system to cover diverse inputs from a user (e.g., how many paw pads does a (cat | kitten | tabby) → (cat) have?). therefore, to deploy hierarchical information in such systems, it is critical to represent word generality and meaning efficiently in a machine-readable manner. for encoding word relationships, distributional representations are commonly used in natural language processing. word embeddings such as word2vec @xcite , glove @xcite , and fasttext @xcite express word similarities as continuous vectors in euclidean space, and have brought significant advances to various applications @xcite . hierarchy-aware representations in euclidean space have also been developed. gaussian embedding @xcite represents words as gaussian distributions, whose mean vectors (50-100 dimension) encode word similarity, and whose variances encode word generality. mean vectors have a similar role to existing word vectors, and word generalities are expressed by adding variance. on the other hand, order embedding @xcite expresses both generality and similarity using word vectors, namely word positions. it models the partially or-dered structure of a hierarchy between objects as inclusive relations of orthants in euclidean space. more recently, models using non-euclidean space -hyperbolic, spherical space, or space of heterogeneous curvature have been gaining researchers' attention @xcite @xcite @xcite . among them, hyperbolic geometry has good compatibility with spreadingout structures of hierarchies, since the volume of hyperbolic space increases exponentially to the direction of the radii @xcite . for instance, poincaré embedding @xcite , lorentz embedding @xcite , and hyperbolic cone @xcite perform excellently even with low dimensions by utilizing hyperbolic space. the idea of poincaré embedding representing a hierarchy with a ball is intuitive and promising. the model learns embeddings of which (i) the distance from the origin of the ball represents generality of objects (e.g., mammal and dog) and (ii) the difference of angles represents similarity between objects (e.g., dog and cat), as shown in figure 1 . however, hyperbolic embeddings require other components to also be developed under the same hyperbolic geometry @xcite . it may be challenging to apply the model to downstream applications that are mostly developed in euclidean space @xcite . our goal is to achieve a low-dimensional, intuitive representation of hierarchical structures, such as poincare embedding, in commonly used euclidean space. in this paper, we propose polar embedding for learning representations on the polar coordinate system. polar embedding can show word hierarchies explicitly using two independent variables: word generality is expressed by radius, and word similarity is expressed by angles. radius and angles can be optimized separately, which allows us to use beneficial resources such as word frequencies or word generality annotations for computing radiuses. in short, polar coordinates provide us with a useful system to achieve the intuitive distribution of a hierarchy. we also introduce techniques for learning hierarchy-aware representations while efficiently using an area in low-dimensional euclidean space with the polar coordinate system. to sum up, the contributions of this paper are threefold; 1. we introduce polar coordinates to learn hierarchy-aware embedding in euclidean space. ## related work popular word embeddings train word vectors by minimizing the euclidean distance between words appearing in the same context @xcite . word embeddings showed significant progress in numerous nlp research; however, those embeddings have two points to be improved regarding the vector norm. @xcite pointed out the first problem that there is a gap between training space and usage space of word embedding. the distributed representation is trained to minimize the distance between vectors of similar words. although, the word vectors are normalized to the unit length and measured word similarities using cosine distance @xcite . the usage space is a ball, which is different from the training space. to learn a distributed representation in a space suitable for evaluation with cosine similarity, @xcite proposed spherical text embedding that use riemannian manifold to learn word vectors with unit-norm constraints. their model achieves high performance in the document classification task by learning embeddings in the same space as the usage space. an other problem of popular word embeddings is that there is no explicit modeling of hierarchical relationships. subsequent studies have shed light on the issue and extended word embeddings to be aware of hierarchical information. gaussian embedding @xcite considers a hierarchy as an inclusion relation and represents it as gaussian distributions with different variances so that general words have higher variance. word norms are frequently utilized to represent hierarchical structures, because in popular word embeddings such as glove, word vectors are normalized and norms are not utilized effectively. @xcite introduced a loss function to reflect pairwise hypernymy relations in similarity of word vectors, and @xcite proposed a post-processing method for adjusting vector norms to enhance hierarchical relationships. order embedding @xcite represents a hierarchy by preserving the partial order between words. by learning similarity on the euclidean sphere and by expressing word generality in norm, it is expected that the word hierarchy can be effectively represented in euclidean space. ## polar embedding we propose polar embedding that learns word representations in the polar coordinate system. the most essential feature of polar coordinates is that it holds the radius and angles of a position vector as separate parameters. given that the intuitive distribution of a hierarchical structure shown in figure 1 , we can naturally associate the radius with word generality and the angles with word similarity. in this section, we describe methods of optimizing the radius and angles towards the simple but efficient representation of a hierarchy in low-dimensional euclidean space. ## experiments following previous studies @xcite , we used the transitive closure in wordnet @xcite as our experimental dataset and compared polar embedding with algorithm 1 learning procedure of angles input for the main loop: iteration n , dataset d, vocabulary v , learning rate α, weight for negative samples β, svgd interval s input for svgd: iteration m , learning rate η, early stopping criterion γ ∈ [0, 1] 1: p θ , p ϕ k ← approximate angle uniform distributions on a sphere with gmm 2: θ, ϕ k ← initialize vectors in cartesian coordinates with a normal distribution, then convert them into polar coordinates for starting with a uniform distribution on a sphere @xcite 3: for @xmath0: w (t) , w (+) , w (-) ← sample from d 5: 6: // update of θ with the welsch loss 7: no update if φk w (t) overflows from the range 13: 14: // update of θ and ϕ k (∀@xmath1, n -2}) with svgd 15: for w @xmath2: x ← create a set of batched samples from x 24: ## conclusion we proposed polar embedding, which represents hierarchical structures in low-dimensional euclidean space. word generalities and similarities are intuitively expressed using radius and angles in polar coordinates. we introduced the welsch loss function and svgd for training embeddings in the angle limit of polar coordinates, which keeps angle distributions uniform and enables a model to leverage a whole space effectively. experimental results indicated that polar embedding outperformed other embeddings in euclidean space.