Synthetic, yet natural: Properties of W ord N et random walk corpora and the impact of rare words on embedding performance (2019)

Abstract: Creating word embeddings that reflect semantic relationships encoded in lexical knowledge resources is an open challenge. One approach is to use a random walk over a knowledge graph to generate a pseudo-corpus and use this corpus to train embeddings. However, the effect of the shape of the knowledge graph on the generated pseudo-corpora, and on the resulting word embeddings, has not been studied. To explore this, we use English WordNet, constrained to the taxonomic (tree-like) portion of the graph, as a case study. We investigate the properties of the generated pseudo-corpora, and their impact on the resulting embeddings. We find that the distributions in the psuedo-corpora exhibit properties found in natural corpora, such as Zipf’s and Heaps’ law, and also observe that the proportion of rare words in a pseudo-corpus affects the performance of its embeddings on word similarity.

## introduction a word embedding model maps the words in a vocabulary to dense low-dimensional vectors, by inferring the relative position of each word in a shared multidimensional semantic space from its context of use in a corpus @xcite . this approach is founded on the distributional hypothesis @xcite , which states that words which occur in the same contexts tend to have similar meanings. such word embeddings are created by training a neural network language model on natural language corpora. while such embeddings have been shown to perform well on semantic relatedness benchmarks @xcite , training on a natural corpus only models one type of semantic relation between words: thematic (i.e. syntagmatic). on the flip side, taxonomic (i.e. paradigmatic) relations are not explicitly contained in natural language corpora, and as such are not included in those embeddings @xcite . in fact, research suggests that the best measures of taxonomic similarity and thematic relatedness are different in distributional space @xcite . furthermore, there are many other kinds of relationships between words and concepts that can be found in knowledge engineered resources, such as knowledge bases, ontologies, taxonomies and other semantic networks. modelling these relations is an important task in building ai with comprehensive natural language understanding abilities, and there have been many efforts to bring knowledge graphs into an embedding space (see section 2 for details). one such approach is the wordnet random walk algorithm @xcite : by randomly walking the wordnet knowledge graph and choosing words from each synset that has been traversed, a pseudo-corpus is generated and used for training word embeddings. the reasoning is that the distributional hypothesis should also apply in this scenario, in the sense that co-occurrence within local contexts in the pseudo-corpus will reflect the connections between words connected in the wordnet graph. naturally, the shape of the underlying knowledge graph (in terms of node connectivity: i.e. tree, fully-connected, radial etc.) affects the properties of a pseudo-corpus generated via a random walk over the graph. developing a better understanding of the relationship between the shape of a knowledge graph, the properties of the resulting pseudo-corpora, and the properties of the resulting embeddings, has the potential to inform how the walk over a given knowledge graph should be tailored to improve embedding performance. in this paper we provide an analysis of some of the properties of pseudo-corpora generated using the random walk method, and examine the impact of these properties on embedding performance. we base this analysis on the wordnet taxonomy, because (a) wordnet is one of the mostpopular taxonomies in use, and (b) in general, the wordnet taxonomy has a well-understood shape (tree-like) which informs the analysis of our results. we find that the pseudo-corpora synthesized from the wordnet taxonomy are not as artificial as one might expect -they exhibit properties and regularities also found in natural corpora, following natural language laws such as heaps' law and zipf's law. consequently, we hypothesise that word embeddings trained on such corpora might face the same limitations as those trained on natural corpora would. we explore this notion on the case study of rare (i.e. infrequent) words, which are a known problem for word embeddings @xcite @xcite . ## related work research on building embeddings from knowledge resources such as wordnet @xcite , can be broadly categorised into three approaches: i) enrichment, ii) specialisation, and iii) direct learning from knowledge resources. both enrichment and specialisation modify precomputed, corpus-based word embeddings with information from a knowledge resource to either augment them (enrichment) or to fit them onto the specific semantic relation described by that knowledge resource (specialisation). retrofitting @xcite is an example of enrichment: it modifies corpus-based embeddings by reducing the distance between words that are directly linked in resources like wordnet, mesh @xcite and conceptnet @xcite . in our own recent related work, we have explored the impact of corpus size on vector enrichment @xcite . on the other hand, examples of the specialisation approach are paragram @xcite , attract-repel @xcite , @xcite and the work of @xcite and @xcite on synonyms and antonyms. @xcite and @xcite introduce global specialisation models where vectors for words that are missing in the knowledge resource are also updated. more related to our work are the approaches to learn directly from knowledge resources. examples include building non-distributional sparse word vectors from lexical resources @xcite , building poincaré embeddings that represent the structure of the wordnet taxonomy @xcite and building embeddings that encode all semantic relationships expressed in a biomedical ontology within a single vector space @xcite . the latter two methods encode the semantic structure of a knowledge resource in a deterministic manner, while @xcite follow a stochastic approach based on personalised pagerank: they compute the probability of reaching a synset from a target word, following a random-walk on a given wordnet relation. instead of computing random-walk probabilities, @xcite use an off-theshelf implementation of the word2vec skip-gram algorithm to train embeddings on wordnet random walk pseudo-corpora, changing neither the embedding algorithm nor the objective function foot_0 . the resulting embeddings encode wordnet taxonomic information rather than natural word cooccurence. an advantage of the embeddings produced by this method is that they can be used as is or can be combined with real-corpus embeddings in order to accomplish enrichment or specialisation @xcite . previous work has analysed semantic properties of word embeddings generated by random walk. @xcite , for example, found wordnet random-walk embeddings to outperform corpus-based word embeddings on the strict semantic similarity (taxonomic similarity) simlex-999 benchmark @xcite , confirming that they encode taxonomic information better than real-corpus word embeddings. additionally, other researchers have explored different varieties of the random walk algorithm. most notably, @xcite drastically enrich the graph structure by using all available relationships between wordnet synsets, while inferring and adding others from outside resources @xcite . however, to the best of our knowledge, there has been no work on analysing the properties of the corpora generated by random-walk processes. in particular, there has been no work on comparing their statistical properties with those of natural corpora. ## evaluation and analysis after generating all the corpora, we trained word embeddings and evaluated their performance on the task of word similarity. ## conclusion in our work we expand our understanding of the random walk algorithm, in terms of the relationship between the structure of the underlying knowledge graph, the properties of the pseudocorpora generated from the graph, and the performance of the embeddings trained on these pseduo-corpora. we use the wordnet taxonomy as a case study for our work. we find that all our pseudocorpora resemble natural corpora at a statistical level. we attribute these properties to the underlying tree structure of the graph from which the pseudo-corpora are built. we also train word embeddings on these corpora to study the impact of these properties on the embedding performance on word similarity evaluation tasks. our evaluations confirm a successful modelling of taxonomic relations, and on most benchmarks our data supports the hypothesis that the ratio of rare words in a pseudo-corpus affects embedding performance. understanding the properties of the pseudocorpora generated from a knowledge graph structure can inform how the random walk should be designed and run for any graph. e.g. knowing that a tree-like graph structure results in pseudocorpora exhibiting zipfian properties is useful as it highlights the presence of rare words in the corpora. as the vocabulary of the lexical resource is finite, the problem of rare words within the generated pseudo-corpora can be addressed by ensuring that the pseudo-corpus is large enough so that even the relatively rare words appear frequently enough to learn adequate embeddings. this perspective helps in answering questions such as: how large should a pseudo-corpus be? though this might seem obvious, an important takeaway is that the properties of any pseudocorpus generated from a knowledge graph will be affected by the properties of that graph-its structure and node connectivity will be reflected in the generated corpora, thus impacting the resulting embeddings. we do not claim that any graph structure will exhibit the exact properties we found, but rather that this kind of analysis should be considered when using a random walk algorithm. as far as future work, there are several exciting avenues that can be explored. most immediately, it would be important to examine whether the natural properties and rare word percentages in the pseudo-corpora hold when applied to more dense graph structures with connections beyond the wordnet taxonomy, such as wordnet gloss relations, polysemy, antonymy, meronymy, etc. going further, one could apply the random walk to other knowledge bases to see if the regularities hold there as well. additionally, combining pseudo-corpora from different knowledge bases, or simply enriching one graph with connections from another, adding additional thematic relations from other knowledge bases. certainly, this would make the problem more complex, and would render the directionality parameter moot, as a lot of those connections do not have an inherent directionality to them. but this is definitely the next step in improving scores and increasing coverage. going even further, it would be beneficial to explore the application of both these taxonomic embeddings, as well as more complex knowledge graph embeddings, on tasks other than word similarity, such as hypernym prediction (which are better suited to exploiting taxonomic knowledge) or perhaps using them to tackle the problem of type and token identification of multi-word expressions.