Evaluation of Taxonomy Enrichment on Diachronic W ord N et Versions (2021)

Abstract: The vast majority of the existing approaches for taxonomy enrichment apply word embeddings as they have proven to accumulate contexts (in a broad sense) extracted from texts which are sufficient for attaching orphan words to the taxonomy. On the other hand, apart from being large lexical and semantic resources, taxonomies are graph structures. Combining word embeddings with graph structure of taxonomy could be of use for predicting taxonomic relations. In this paper we compare several approaches for attaching new words to the existing taxonomy which are based on the graph representations with the one that relies on fastText embeddings. We test all methods on Russian and English datasets, but they could be also applied to other wordnets and languages.

## introduction taxonomic structures are often used for the downstream tasks like lexical entailment @xcite , entity linking @xcite , named entity recognition @xcite . therefore, they always need to be up-todate and to keep up with the language change. moreover, with the rapid growth of lexical resources for specific domains it becomes more and more important to develop systems that could automatically enrich the existing knowledge bases with new words or at least facilitate the manual taxonomy extension process. in this paper we tackle the taxonomy enrichment task which aims at associating new words (words not present in a taxonomy) with the appropriate hypernym synsets from the taxonomy. for instance, the word "foster-child" should be attached to the hypernym synset "child.n.1" (which refers to "child", "kid", "youngster") from word-net, and the word "cactus" -to the synset "succulent.n.1". a word may have multiple hypernyms. the task of finding a single suitable synset is difficult for a machine, and a model trained to solve this task will inevitably return many false answers if asked to provide only one synset candidate. on the other hand, if we relax the requirement of uniqueness and ask instead to provide n (for example, 10 or 15) most suitable candidates, this list can contain correct synsets with higher probability. this setting is also suitable for the manual annotation: presenting an annotator with a small list of candidates will facilitate the annotation process, because the annotator will not need to look through all synsets of the taxonomy. thus, the task is usually formulated as the soft ranking problem, where we need to rank all the synsets according to their suitability for a given word. while word embeddings demonstrate decent results for predicting hypernyms @xcite , much less attention is paid to the approaches based on graph representations. we assume that graph-based representations are complementary to the distributional word embeddings, as they capture the hypo-hypernymy relations from graphs. we expect that models using graph representations could be beneficial for the taxonomy enrichment task in combination with distributed word vector representations or on their own. we check our hypothesis on several models which make use of graph structures: node2vec @xcite , poincaré embeddings @xcite and @xcite and compare it with an approach of @xcite which applies fasttext @xcite and features from wiktionary. all in all, our contribution is the exploration of graph-based representation for the taxonomy enrichment task and its combination with the word distributed representations. the existing studies on the taxonomies can be divided into three groups. the first one addresses the hypernym discovery problem @xcite : given a word and a text corpus, the task is to identify hypernyms in the text. however, in this task the participants are not given any predefined taxonomy to rely on. the second group of works tackles taxonomy induction problem @xcite @xcite , where the goal is to create a taxonomy automatically from scratch. the third group deals with the taxonomy enrichment task: the participants need to extend a given taxonomy with new words @xcite . both word and graph representations can be applied to any of these tasks. ## diachronic wordnet datasets for this task we use two diachronic datasets described by @xcite : one for english, another one for russian based respectively on princeton wordnet @xcite and ru-wordnet taxonomies. each dataset consists of a taxonomy and a set of novel words to be added to this resource. the statistics are provided in ## taxonomy enrichment methods we test a number of methods that make use of taxonomy structure to predict hypernyms for the unseen words and compare their performance with the existing approach that is based on fasttext embeddings. we describe each method in the corresponding section. ## experiments in this section, we report the performance of our models on the taxonomy enrichment task and discuss reasons of low performance of methods exploiting hierarchical structure of the taxonomy. ## conclusion in this work, we experimented with the graphbased representations for the taxonomy enrichment task and compared them to word vector representations. we tested approaches based on poincaré and node2vec embeddings along with the approach based on graph autoencoder to predict hypernym synsets for the input word. our results show that the use of word vector representations is much more efficient than any of the tested graph-based approaches. moreover, our baseline method (candidates retrieved from fast-text nearest neighbour list and ranked with features extracted from wiktionary) does not benefit from graph-based methods. namely, combining the baseline scoring function with poincaré and node2vec similarities results in marginal improvements for some datasets, but this does not hold for all of them. according to our experiments, word vector representations are simple, powerful, and extremely effective instrument for taxonomy enrichment, as the contexts (in a broad sense) extracted from the pre-trained fasttext embeddings are sufficient to attach new words to the taxonomy. error analysis also reveals that the correct synsets identified by graph-based models are usually retrieved by the fasttext-based model alone. this makes graphs representations irrelevant and excessive. nonetheless, there exist cases where graph representations were able to identify correctly some hypernyms which were not captured by fasttext. despite the discouraging first results of the application of graph-based methods, we suggest that the taxonomy enrichment task could still benefit from them. in order to improve their performance, we plan to switch from linear transformation to non-linear to project fasttext embeddings to node2vec and to apply recently published unsupervised graph word representations graph-glove @xcite . moreover, we find it promising to experiment with temporal embeddings such of those of @xcite for the taxonomy enrichment task.