Evaluating Natural Alpha Embeddings on Intrinsic and Extrinsic Tasks (2020)

Abstract: Skip-Gram is a simple, but effective, model to learn a word embedding mapping by estimating a conditional probability distribution for each word of the dictionary. In the context of Information Geometry, these distributions form a Riemannian statistical manifold, where word embeddings are interpreted as vectors in the tangent bundle of the manifold. In this paper we show how the choice of the geometry on the manifold allows impacts on the performances both on intrinsic and extrinsic tasks, in function of a deformation parameter alpha.

## introduction word embeddings are compact representations for the words of a dictionary. @xcite first introduced the idea of using the internal representation of a neural network to construct a word embedding. @xcite employ a neural network to predict the probability of the next word given the previous ones. @xcite proposed the use of a recurrency language model based on rnn, to learn the vector representations. more recently, this approach has been exploited further, with great success by means of bidirectional lstm @xcite and transformers @xcite @xcite . in this paper we focus on skip-gram (sg), a well-known model for the conditional probability of the context of a given central word, which it has been shown to work well at efficiently capturing syntactic and semantic information. sg is at the basis of many popular word embeddings algorithms, such as word2vec @xcite , the contpdfinfoinuous bag of words @xcite , and models based on weighted matrix factorization of the global cooccurrences as glove @xcite , cf. @xcite . these methods are deeply related, levy and goldberg showed how word2vec sg with negative sampling is effectively performing a matrix factorization of the shifted positive pmi @xcite . it has been noted @xcite how, once the embedding space has been learned, syntactic and semantic analogies between words translate in linear relations between the respective word vectors. there have been numerous works investigating the reason of the correspondence between linear properties and word relations. @xcite gave a very intuitive explanation in their paper on glove @xcite . more recently arora et al. @xcite tried to study this property by introducing a hidden markov model, under some regularity assumptions on the distribution of the word embedding vectors, cf. @xcite . word embeddings are also often used as input for another computational model, to solve more complex inference tasks. the evaluation of the quality of a word embedding, which ideally should encode syntactic and semantic information, is not easy to be determined and different approaches have been proposed in the literature. this evaluation can be in terms of performance on intrinsic tasks like word similarity @xcite @xcite , or by solving word analogies @xcite , however several authors @xcite has showed a low degree of correlation between the quality of an embedding for word similarities and analogies on one side, and on downstream (extrinsic) tasks, for instance on classification or prediction, to which the embedding is given in input. several works have highlighted the effectiveness of post-processing techniques @xcite , such as pca @xcite , focusing on the fact that certain dominant components are not carriers of semantic nor syn-tactic information and thus act like noise for determinate tasks of interest. a different approach which still acts on the learned vectors after training has been recently proposed by @xcite . the authors present a geometrical framework in which word embeddings are represented as vectors in the tangent space of a probability simplex. a family of word embeddings called natural alpha embeddings is introduced, where α is a deformation parameter for the geometry of the probability simplex, known in information geometry in the context of α-connections @xcite . noticeably, alpha word embeddings include the classical word embeddings as a special case. in this paper we provide an experimental evaluation of natural alpha embeddings over different tasks, both intrinsic and extrinsic, including word similarities and analogies, as well as downstream tasks, such as document classification and sentiment analysis, in order to study the impact of the geometry on performances. ## conditional models and the embeddings structure the skip-gram conditional model @xcite allows the unsupervised training of a set of word-embeddings, by predicting the conditional probability of any word χ to be in the context of a central word w with z @xmath0. the conditional model represents an exponential family in the simplex, parameterized by two matrices u and v of size n×d, where n is the cardinality of the dictionary d, and d is the size of the embeddings. we will refer to the rows of a matrix v as v χ or v χ , and to its columns as v k . it is common practice in the literature of word embedding to consider u w or alternatively u w + v w as embedding vectors for w @xcite @xcite . in the remaining part of this section we briefly review the natural alpha embeddings and limit embeddings, based on information geometry framework. we refer the reader to @xcite for more details and mathematical derivations. ## experiments we considered two corpora: english wikipedia dump october 2017 (enwiki), with 1.5b words, and its augmented version composed by gutenberg (gutenberg), english wikipedia and book-corpus @xcite ; bookcorpus; kobayashi) (geb), with 1.8b words. for each corpus we trained a set of glove word embeddings @xcite with vector sizes of 300 and 50, window size of 10, until convergence for a maximum of 1,000 epochs (more details in appendix a). the embeddings in eq. ( 2 ) will be denoted with 'e' in figures and tables, while the limit embeddings in eq. ( 9 ) will be denoted with 'le'. embeddings have been normalized either with the fisher information matrix (f) or with the identity (i). similarly after normalization, the scalar products can be computed with the respective metric (on the tasks that requires scalar product calculation). in this study, normalization and scalar product are always using the same metric. for the reference distribution needed for the computation of the alpha embeddings we have chosen the uniform distribution (0), the unigram distribution of the model (u) -obtained by marginalization of the joint distribution learned by the model, or the unigram distribution estimated from the corpus data (ud). embeddings are denoted by 'u', if in the computation of eqs. (2) and ( 9 ), the formula used for p w is eq. ( 1 ), while they will be denoted by 'u+v' if eq. ( 7 ) is used instead. we evaluated the alpha embeddings on intrinsic (similarities, analogies, concept categorization) and extrinsic (document classification, sentiment analysis) tasks. ## conclusions for word similarities and analogies alpha embeddings provide significant improvements over baseline methods (corresponding to α = 1). for the other tasks the improvements are smaller but consistent, depending on the value of α, the chosen reference distribution (0, u, ud) and the chosen normalization method (i, f). the improvements persist also when increasing the complexity of the networks used (linear vs bilstm). this motivates further studies on more complex architectures, for example on models employing transformers with the aim to close the experimental gap with the state of the art. the best value of alpha depends both on the task and on the dataset. alpha embeddings thus provide an extra handle on the optimization problem, allowing to choose the deformation parameter based on data. alpha values lower than 1 and negative seems to be preferred across most tasks. limit embeddings provide a simple method which does not require validation over alpha, but can still offer an improvement on several tasks of interest. furthermore limit embeddings can be interpreted as a natural clustering in space learned by the sg model itself during training. performances of the limit embeddings grow with increasing dimension, pointing to the possibility to have a consistent improvement in higher embedding dimensions without tuning alpha.