B ayesian Hierarchical Words Representation Learning (2020)

Abstract: This paper presents the Bayesian Hierarchical Words Representation (BHWR) learning algorithm. BHWR facilitates Variational Bayes word representation learning combined with semantic taxonomy modeling via hierarchical priors. By propagating relevant information between related words, BHWR utilizes the taxonomy to improve the quality of such representations. Evaluation of several linguistic datasets demonstrates the advantages of BHWR over suitable alternatives that facilitate Bayesian modeling with or without semantic priors. Finally, we further show that BHWR produces better representations for rare words.

## introduction in the last decade, a plethora of methods were proposed for learning vector representations for words @xcite @xcite , sentences @xcite , items @xcite @xcite , and medical concepts @xcite . in the domain of natural language understanding, neural word embedding models are designed to learn distributed word representations as vectors in a latent space. in this space, arithmetic operations between the word vectors encode semantic and syntactic information. specifically, the seminal works by @xcite @xcite , exhibited state-of-the-art performance on various linguistic tasks @xcite @xcite . the major focus of most previous models was on optimizing the utilization of co-occurrence relations for learning representations, e.g., learning the probability of word 𝑥 to appear in the vicinity of word 𝑦 . yet, often, additional side information can be leveraged for learning finer embeddings. in this work, we focus on incorporating word semantic taxonomy, which is particularly useful for learning representations of rare words and for learning word representations from a small-size corpus. to this end, we introduce the bayesian hierarchical words representation (bhwr) learning algorithm. bhwr presents two complementary properties: bayesian modeling of word representations aside with hierarchical priors that naturally support semantic taxonomy. bhwr is based on a variational bayes (vb) optimization that enables the mapping of words into probability densities in the latent space. a key advantage of bhwr is the utilization of word taxonomy for the propagation of relevant information between related words. for example, consider the words 'anode' and 'cathode'. both words have a common relationship to the word ## related work incorporating lexical-semantic information in learning word embeddings has been suggested in the past. in @xcite , a post-processing technique was introduced in order to refine pretrained word representations using relational information from semantic lexicons. in @xcite , hierarchical taxonomy was utilized for improving document categorization and concept clustering. recently, linguistic knowledge bases were utilized for enhancing contextualized word embeddings @xcite . the abovementioned works finetune pre-trained representations, while injecting external contextual information for words (some w.r.t. a specific task). unlike these works, bhwr facilitates bayesian learning of non-contextualized word embeddings, in combination with hierarchical taxonomy information which is not task-specific. hence, a direct comparison between bhwr and these works is unfitting. more relevant to our work is the bayesian skip-gram (bsg) model from @xcite . however, bsg does not allow the use of external information such as word taxonomy. hence, in our experiments, we compared bhwr to bsg and further apply the method from @xcite to enhance bsg with word taxonomy information. ## bayesian hierarchical words representation in this section, we describe the model and derive a vb solution that is finally translated to the bayesian hierarchical words representation (bhwr) learning algorithm. ## experimental setup and results the experimentations in this section are focused on word similarity. next, we present the training corpus, evaluated models, evaluation tasks, and the results. ## conclusion and future work we presented bhwr -a word representation learning model, facilitating bayesian learning of co-occurrences relations together with word taxonomy via hierarchical priors. when trained on a small corpus, bhwr exhibits a significant performance gain over other word embedding methods across various word similarity datasets. importantly, a remarkable improvement is obtained for rare words. moreover, bhwr outperforms all other baselines even when the latter are enhanced with the post-processing taxonomy refinement procedure from @xcite . finally, when combining bhwr with the post-processing from @xcite , further improvement is observed. in the future, we plan to extend the applicability of the presented model to other linguistics tasks as well as recommendations and medical inference tasks.