Two experiments for embedding W ordnet hierarchy into vector spaces (2019)

Abstract: In this paper, we investigate mapping of the WORDNET hyponymy relation to feature vectors. Our aim is to model lexical knowledge in such a way that it can be used as input in generic machine-learning models, such as phrase entailment predictors. We propose two models. The first one leverages an existing mapping of words to feature vectors (fastText), and attempts to classify such vectors as within or outside of each class. The second model is fully supervised, using solely WORDNET as a ground truth. It maps each concept to an interval or a disjunction thereof. The first model approaches but not quite attain state of the art performance. The second model can achieve near-perfect accuracy.

## introduction distributional encoding of word meanings from large corpora @xcite @xcite have been found to be useful for a number of nlp tasks. while the major goal of distributional approaches is to identify distributional patterns of words and word sequences, they have even found use in tasks that require modeling more fine-grained relations between words than cooccurrence in word sequences. but distributional word embeddings are not easy to map onto ontological relations or vice-versa. we consider in this paper the hyponymy relation, also called the is-a relation, which is one of the most fundamental ontological relations. we take as the source of truth for hyponymy @xcite , which has been designed to include various kinds of lexical relations between words, phrases, etc. however, wordnet has a fundamentally symbolic representation, which cannot be readily used as input to neural nlp models. several authors have proposed to encode hyponymy relations in feature vectors @xcite @xcite . however, there does not seem to be a common consensus on the underlying properties of such encodings. in this paper, we aim to fill this gap and clearly characterize the properties that such an embedding should have. we additionally propose two baseline models approaching these properties: a simple mapping of fasttext embeddings to the wordnet hyponymy relation, and a (fully supervised) encoding of this relation in feature vectors. ## goals we want to model the hyponymy relation (ground truth) given by wordnet -hereafter referred to as hyponymy. in this section we make this goal precise and formal. hyponymy can in general relate common noun phrases, verb phrases or any predicative phrase, but hereafter we abstract from all this and simply write "word" for this underlying set. in this paper, we write (⊆) for the reflexive transitive closure of the hyponymy relation (ground truth), and (⊆ m ) for relation predicted by a model m . 1 ideally, we want the model to be sound and complete with respect to the ground truth. however, a machine-learned model will typically only approach those properties to a certain level, so the usual relaxations are made: property 1 (partial soundness) a model m is partially sound with precision α iff., for a proportion α of the pairs of words w, w such that w ⊆ m w holds, w ⊆ w holds as well. property 2 (partial completeness) a model m is partially complete with recall α iff., for a proportion α of the pairs of words w, w such that w ⊆ w holds, then w ⊆ m w holds as well. these properties do not constrain the way the relation (⊆ m ) is generated from a feature space. however, a satisfying way to generate the inclusion relation is by associating a subset of the vector space to each predicate, and leverage the inclusion from the feature space. concretely, the mapping of words to subsets is done by a function p such that, given a word w and a feature vector x, p (w, x) indicates if the word w applies to a situation (state of the world, sentence meaning, sentory input, etc.) described by feature vector x. we will refer to p as a classifier. the inclusion model is then fully characterized by p , so we can denote it as such (⊆ p ). ## inclusion of subsets a strict interpretation of property 3 would dictate to check if the subsets defined in the previous section are included in each other or not. however, there are several problems with this approach. to begin, hyperplanes defined by θ and b will (stochastically) always intersect therefore one must take into account the actual density of the fasttext embeddings. one possible approximation would be that they are within a ball of certain radius around the origin. however, this assumption is incorrect: modeling the density is a hard problem in itself. in fact, the density of word vectors is so low (due to the high dimensionality of the space) that the question may not make sense. therefore, we refrain from making any conclusion on the inclusion relation of the subsets, and fall back to a more experimental approach. thus, we will test the suitability of the learned p (w) by testing whether elements of its subclasses are contained in the superclass. that is, we define the following quantity q(w , w) = average{p (w , x) | @xmath0, p (w, f (x))} which is the proportion of elements of w that are found in w. this value corresponds to the relaxation parameter ρ in property 4. if w ⊆ w holds, then we want q(w , w) to be close to 1, and close to 0 if w is disjoint from w. we plot (figure 2 ) the distribution of q(w , w) for all pairs w ⊆ w, and a random selection of pairs such that w ⊆ w. the negative pairs are generated by taking all pairs (w , w) such that w ⊆ w, and generate two pairs (w 1 , w) and (w , w 2 ), by picking w 1 and w 2 at random, such that neither of the generated pairs is in the hyponymy relation. we see that most of the density is concentrated at the extrema. thus, the exact choice of ρ has little influence on accuracy for the model. for ρ = 0.5, the recall is 88.8%. the ratio of false positives to the total number of negative test cases is 85.7%. however, we have a very large number of negatives cases (the square of the number of classes, about 7 billions). because of this, we get about 1 billion false positives, and the precision is only 0.07%. regardless, the results are comparable with state-of-the art models (section 6). figure 2 : results of inclusion tests. on the lefthand-side, we show the distribution of correctly identified inclusion relations in function of ρ. on the right-hand-side, we show the distribution of (incorrectly) identified inclusion relations in function of ρ. ## wordnet predicates as disjunction of intervals in this section we propose a baseline, fully supervised model model for hyponymy. the key observation is that most of the hy-ponymy relation fits in a tree. indeed, out of 82115 nouns, 7726 have no hypernym, 72967 have a single hypernym, and 1422 have two hypernyms or more. in fact, by removing only 1461 direct edges, we obtain a tree. the number of edges removed in the transitive closure of the relation varies, depending on which exact edges are removed, but a typical number is 10% of the edges. in other words, when removing edges in such a way, one lowers the recall to about 90%, but the precision remains 100%. indeed, no pair is added to the hyponymy relation. this tree can then be mapped to one-dimensional intervals, by assigning a position to each of the nodes, according to their index in depth-first order (ix(w) below). then, each node is assigned an interval corresponding to the minimum and the maximum position assigned to their leaves. a possible directed acyclic graph (dag) and a corresponding assignment of intervals is shown in fig. 3 . the corre-a b c d e f 1 2 5 3 4 6 [1,6] [3,4] [5,6] [3,3] [4,4] [6,6] a b c d e f 1 2 4 3 5 6 [1,6] [2,3] [4,6] [3,3] [5,5] [6,6] sponding definition of predicates is the following: because it is fully sound, a model like the above can always be combined with another model to improve its recall with no impact on precision -including itself. such a self-combination is useful if one does another choice of removed edges. thus, each word is characterized by an ndimensional co-product (disjoint sum) of intervals. by increasing n, one can increase the recall to obtain a near perfect model. many authors have considered modeling hyponymy. however, in many cases, this task was not the main point of their work, and we feel that the evaluation of the task has often been partially lacking. here, we review several of those and attempt to shed a new light on existing results, based on the properties presented in section 2. several authors @xcite @xcite have proposed feature-vector embeddings of wordnet. among them, several have tested their embedding on the following task: they feed their model with the transitive closure of hy-ponymy, but withhold 4000 edges. they then test how many of those edges can be recovered by their model. they also test how many of 4000 random negative edges are correctly classified. they report the average of those numbers. we reproduce here their results for this task in ## future work and conclusion we found that defining the problem of representing hyponymy in a feature vector is not easy. difficulties include 1. the sparseness of data, 2. whether one wants to base inclusion on an underlying (possibly relaxed) inclusion in the space of vectors, and 3. determining what one should generalize. our investigation of wordnet over fasttext demonstrates that wordnet classes are not cleanly linearly separated in fasttext, but they are sufficiently well separated to give a useful recall for an approximate inclusion property. despite authors result @xcite 90.6 @xcite 92.3 @xcite 92.3 us, fasttext with lr and ρ = 0.5 87.2 us, single interval (tree-model) 94.5 us, interval disjunctions, @xmath1.6 this, and because the negative cases vastly outnumber the positive cases, the rate of false negatives is still too high to give any reasonable precision. one could try to use more complex models, but the sparsity of the data would make such models extremely sensitive to overfitting. our second model takes a wholly different approach: we construct intervals directly from the hyponymy relation. the main advantage of this method is its simplicity and high-accuracy. even with a single dimension it rivals other models. a possible disadvantage is that the multidimensional version of this model requires disjunctions to be performed. such operations are not necessarily available in models which need to make use of the hyponymy relation. at this stage, we make no attempt to match the size of intervals to the probability of a word. we aim to address this issue in future work. finally, one could see our study as a criticism for using wordnet as a natural representative of hyponymy: because wordnet is almost structured like a tree, one can suspect that it in fact misses many hyponymy relations. this would also explain why our simple fasttext-based model predicts more relations than present in word-net. one could think of using other resources, such as jeuxdemots @xcite . yet our preliminary investigations suggest that these suffer from similar flaws -we leave a complete analysis to further work.