Lexical Entailment with Hierarchy Representations by Deep Metric Learning (2022)

Abstract: In this paper, we introduce a novel method for lexical entailment tasks, which detects a hyponym-hypernym relation among words. Existing lexical entailment studies are lacking in generalization performance, as they cannot be applied to words that are not included in the training dataset. Moreover, existing work evaluates the performance by using the dataset that contains words used for training. This study proposes a method that learns a mapping from word embeddings to the hierarchical embeddings in order to predict the hypernymy relations of any input words. To validate the generalization performance, we conduct experiments using a train dataset that does not overlap with the evaluation dataset. As a result, our method achieved state-of-the-art performance and showed robustness for unknown words.

## introduction lexical entailment (le) is a task to predict hypernym-hyponym relationships between two words, such as "a swan is a bird." and organize terms in a hierarchical order. le can be used not only for the construction of thesaurus @xcite , but also for semantic disambiguation @xcite and visualization @xcite of the substantial amount of information extracted from texts. methods utilizing hierarchical word embeddings are the recent mainstream for le, and they can be classified into two categories. one is based on the distributive inclusion hypothesis (dih), which assumes that if the specific word like "swan" is semantically entailed by the more general word like "bird", then the context in which "swan" occurs is relatively less frequent than and is included in the context in which "bird" occurs @xcite . based on dih, @xcite developed lear, a method to post-process the pre-trained distributed representations to obtain embeddings emphasizing hypernymy relations. this type of embedding in euclidian space is easy to apply in multiple ways @xcite , such as visualization of lexicons and cross-lingual word translation @xcite . embedding hierarchical structures in hyperbolic space is the other category. hyperbolic space's characteristic of exponentially increasing volume at points far from the origin is well suited to tree structure with multiple child nodes. thus research on embedding in hyperbolic spaces has been attracting attention in the field of machine learning @xcite and also in le @xcite . another stream is pattern-based approach that examines appearance patterns of hypernymies in large corpus. they began with hearst patterns @xcite , and in recent years, a variety of methods have been proposed, including using the hierarchical attention network @xcite or combining patterns with word embedding operations @xcite . one of the major drawbacks of these methods is that they are not able to predict hypernymies between unknown words, which are not included in the train dataset and/or corpus. existing studies define the vocabulary set in advance and directly learn the distributed representation of each word as a parameter. furthermore, this situation forces the performance evaluation to be laden with leakage. the evaluation datasets generally used in le tasks consist of words contained in a train dataset, such as wordnet @xcite . this makes it impossible to evaluate generalization performance for unknown words properly, which is an undesirable situation in machine learning. considering the real-world application of le, the ability to locate unknown words that correspond to brand-new technologies and concepts at appropriate coordinates is critical. only a few researches have coped with this problem, such as using gans imitating the lear embeddings @xcite , learning word-relation vectors by simulating the hypernymy generation process @xcite , and simply conducting unsupervised learning of glove on a hyperbolic space @xcite . however, the accuracy of the first study is limited by the performance of lear, and the rest have problems with embedding applications. in this paper, we propose a method to obtain a hierarchical embedded representation that can be applied to unknown words, while maintaining the distinctiveness and usability. in order to acquire generalization performance for unknown words, our method is designed to learn the mapping from the pre-trained word vectors to the hierarchical embeddings, instead of hierarchical embeddings themselves. in addition, we aim to obtain representations more discriminative for hypernymy relations by using ranked list loss @xcite , a deep metric learning method, for learning this mapping function. to evaluate the quality of our embeddings, we conducted an evaluation with standard word-level le task datasets, such as bless @xcite . in order to evaluate the generalization performance for unknown words, training data were created with no word overlap with the evaluation dataset. experimental results demonstrate that our method achieved state-of-the-art prediction performance and is robust to unknown words. we also confirm that the embedding is sufficiently expressive even with a low dimensionality of 5. train dataset let b @xmath0, w 1 r ), ..., } be a set of synonym pairs and b l be a set of hyponymhypernym pairs, which are used as train datasets. as described in a previous study @xcite , every pair is extracted from the thesaurus and divided into subsets consisting of k pairs for mini-batch training. particularly in b l , we always assign hyponym as w l , while specifying a hypernym as w r . encoding hierarchy in the next step, initial word embeddings are acquired with fasttext @xcite . as word embeddings are calculated by summarizing the vectors of its subwords, fasttext provides two functions: estimating the representation for unknown words and retraining the model for additional words. then fasttext representaions are transformed into a hierarchy embedding. let f θ : r d f @xmath1. f θ is represented as a fullyconnected neural network (fnn) with an identity function as the activation function in the last layer and relu functions in the others. ## conclusion this study proposed a method for embedding arbitrary input by learning mappings, and evaluated its discriminative and generalization performance. the method combines subword representations, fnn-based mappings, and a deep metric learning technique rll to obtain hierarchical representations. experimental results indicated that rll improved accuracy and fnn-based mapping contributed to generalization performance. moreover, from the subsequent discussion, we found that the obtained embedding representation was sufficiently expressive even at low dimensions. in conclusion, this study confirmed the effectiveness of deep metric learning in acquiring hierarchical embedding representations, succeeded in developing an effective method for extracting entailment relations for arbitrary words, and extended the possibility of application of lexical entailment 1 .