Leveraging W ord N et Paths for Neural Hypernym Prediction (2020)

Abstract: We formulate the problem of hypernym prediction as a sequence generation task, where the sequences are taxonomy paths in WordNet. Our experiments with encoder-decoder models show that training to generate taxonomy paths can improve the performance of direct hypernym prediction. As a simple but powerful model, the hypo2path model achieves state-of-the-art performance, outperforming the best benchmark by 4.11 points in hit-at-one (H@1).

## introduction hypernymy, or the is-a relation, is one of the most important lexical relations. it is used to create taxonomies of terms and it is the main organizational criterion of nouns and verbs in wordnet @xcite . learning hypernymy is also important in practice, as knowing a word's hypernyms gives an approximation of its meaning, and enables inferences in downstream tasks such as question answering and reading comprehension. predicting hypernymy is still a challenging task for word embeddings @xcite @xcite and previous studies have shown that it is more difficult to predict hypernymy than other lexical relations @xcite . hypernymy prediction is often evaluated against a given taxonomy, typically wordnet @xcite . the main hypothesis that we pursue in this paper is that knowledge of this taxonomy, in particular of taxonomy paths, will be helpful for hypernymy prediction. so we introduce two simple encoderdecoder based models for hypernym prediction that make use of information in the full taxonomy paths. there has been much recent work on modeling lexical relations based on distributed representations @xcite @xcite . however, the task formulations and evaluation datasets have differed widely, making it hard to compare different approaches. we focus on evaluating on hypernymy, rather than jointly on many relations, which can mask strong performance differences across relations. we evaluate our encoder-decoder models against several previous models that have not been evaluated in the same setting before. like many other approaches, we use wordnet as the basis for our experiments. we formulate the task as the task of finding the correct point to attach a new node (synset) to the wordnet taxonomy. we build on the existing wn18rr dataset @xcite , but filter its hypernymy pairs to produce wn18rr-hp, a subset that is leak-free with respect to approaches that use taxonomy paths during training, as we do. we find that one of our new models, hyper2path, achieves state-of-the-art performance on hypernym prediction on wn18rr-hp, exceeding the best performance of benchmark models by 4.11 points in accuracy of the highest-ranked prediction (hit-at-one, h@1). in particular, we observe the greatest performance gain in noun hypernym prediction, where it improves over the best benchmark by 5.17 h@1 points. ## hypernym prediction several tasks related to hypernymy have been proposed under different names: extracting is-a relations from text (hypernym discovery) @xcite @xcite , binary classification of whether two given words are in a hypernym relation (hypernym detection) @xcite @xcite , and constructing or extending a taxonomy (taxonomy induction) @xcite . another recently introduced task is hierarchical path completion @xcite , where, given a hypernym path of length 4 from wordnet, the task is to predict the correct hyponym(s). while hypernymy has long been studied in computational lexical semantics, another thread of recent research on hypernymy comes from the literature on knowledge base completion @xcite @xcite . here, hypernymy is considered as one of multiple different semantic relations between two nodes in a graph. extending from this line of research, we also consider the relation prediction task in a semantic graph, but only focus on one relation of interest, hypernymy. like previous work in knowledge base completion @xcite @xcite , we take wordnet as our experimental space, so we learn hypernymy between synsets rather than raw lemmas. a synset is a basic lexical unit in wordnet, defined as a set of lemmas that are synonymous to each other. a synset thus also functions as one of the senses for each of the lemmas in the set. for example, the synset mark.n.01 (a number or letter indicating quality) consists of the three lemmas mark, grade, and score. given a new synset, which we call the source node, our task is to predict its direct hypernym or target node from among the synsets in wordnet. for example, for the source node woolly daisy.n.01, the model should identify wildflower.n.01 as the target node in the graph. while some previous approaches have predicted indirect hypernyms using the transitive closure of wordnet @xcite , we focus on predicting direct hypernyms. datasets for indirect hypernymy often include hypernyms that are too generic and not informative enough, where some semantically distant concepts are trivially mapped to the root node (entity.n.01) or a high-level hypernym near the root. we also restrict ourselves to modeling hypernym relations specifically, unlike much work on wn18rr which learns hypernymy as one of 11 lexical relations @xcite @xcite ). as we discuss in section 5, we observe that hypernymy is more effectively learned when trained on its own. ## models in this section we introduce our two new path-based models, hypo2path and path encoder foot_0 , along with the four benchmark models. ## results and discussion results overall, hypo2path rev shows the highest aggregate (micro-averaged) h@1 (dev: 24.43, test: 25.59) across the three hypernymy relations (nouns, instance nouns, and verbs), while crim has the best aggregate ballpark correctness (wup) scores that are closely followed by hypo2path rev ( for both nouns ( for nouns, the reversed hypo2path model ('hypo2path rev') achieved the best performance on h@1, while the non-reversed and reversed versions performed similarly in terms of wup scores. without reversing the path, the model's h@1 slightly degraded, and is closer to hypo2hyper's results. the performances of path encoder followed closely after the proposed three encoder-decoder models. on instance nouns, performances of different models are overall much higher than on nouns. the best results are observed from the hypo2hyper, hypo2path rev, and text2edges models. that hypo2hyper and hypo2path rev perform similarly is not surprising: instance hypernymy is less likely to be learned from path generation, as it is a special type of hypernymy that only holds between a leaf and its parent node. on the other hand, none of the models foot_15 does well on verb hypernymy ( discussion despite being trained with about ten times less data, scores for instance hypernymy are generally much higher than for noun hypernymy. our finding that it is easier to predict hypernyms for individual entities than for common nouns is consistent with previous work @xcite @xcite . on the other hand, scores for hypernymy amongst verbs are very low, despite having about twice as much training data as instance hypernyms. this could be due both to the fact that verbs have more examples error type query hyponym gold hypernym prediction confounding hyponym % indirect hypernym chant.n.01 religious song.n.01 music.n.01 14.4 folk singer.n.01 singer.n.01 musician.n.01 co-hyponym tart.n.03 pastry.n.02 pie.n.01 3.6 knitwear.n.01 clothing.n.01 apparel.n.01 polysemy nut.n.03 [nut] block.n.01 seed.n.01 nut.n.01 [nut] 17.4 (shared lemmas) chest.n.02 [chest] box.n.01 external body part.n.01 breast.n.01 [breast, chest] multi-word lemmas night porter.n.01 doorkeeper.n.03 evening.n.01 guest night.n.01 15.4 with shared words [night porter] [guest night] carpet beater.n.01 beater.n.02 cleaning implement.n.01 carpet sweeper.n.01 [carpet beater, rug beater] [carpet sweeper, sweeper] our results also suggest that it may be more effective to learn hypernymy separately from other lexical relations, rather than in a multi-relational setup. for example, m3gm trained with all 11 relations of wn18rr achieved a high aggregate validation h@1 of 39.88 in our replication , but a much lower h@1 for hypernymy (1.19) and instance hypernymy (3.74). these were evaluated on the original wn18rr, following @xcite , so these scores are not comparable to tables 3 , 4 , 5, 6. unlike pinter and eisenstein (2018), we evaluated m3gm in one direction (i.e., only hypernym prediction, rather than both hyponym and hypernym prediction). 17 these results are in line with the h@10 scores reported by @xcite and @xcite , which are substantially lower for hypernymy than for the other lexical relations in wn18rr. ## error analysis here we examine the predictions of the best performing model, hypo2path rev, on the validation set for nouns in wn18rr-hp (647 synsets). path validity regardless of whether the predicted direct hypernym was correct or not, we observe that every generated path, from each predicted hypernym to the root, is actually a valid path in wordnet. this is not surprising, since all such paths appear in the training data. while the model always correctly generated valid paths in the graph, they did not necessarily start at the correct node (i.e., failing to predict the gold direct hypernym). nearby nodes for noun hypernymy, 14.4% of the errors are due to predicting an indirect hypernym ( similar synset embeddings some synsets were similar enough to mislead the model: polysemous lemmas were shared across different synsets (17.4% of the total errors) and some multi-word lemmas had shared words (15.4%) ( ## conclusion and future work in this paper we have considered the hypernym prediction task, the task of identifying the correct direct hypernym in a taxonomy of a synset given its embedding. in terms of evaluation, we have focused on both "exact match" (h@1) and "ballpark match" (wup) metrics, and we have for the first time provided a comparison of existing models that had not previously been evaluated with the same metrics or on the same datasets. we have introduced two simple encoder-decoder based models for hypernym prediction that make use of information in the full taxonomy paths, finding that in particular hypo2path rev shows state-of-the-art performance on the wn18rr-hp dataset. for nouns, it achieves improvements of 5.17 test h@1 over the best benchmark model. for verbs, we find that no model achieves a high performance. instance nouns, on the other hand, are the easiest to predict. encouragingly, we find that "ballpark match" (wup) for instance nouns is at over 87 points. there are several directions for future work. encoding lemmas separately and with attention, rather than with a single embedding, could allow the attention mechanism to assign lower weights to less informative, misleading, or polysemous lemma names (which were related to over a third of all errors). one potential way to handle polysemy could involve encoding both synsets and glosses, using either classical word embeddings or contextualized word embeddings. another extension of this work is to use multi-task learning with multiple decoders and different tasks, similar to @xcite . for example, in addition to learning to decode a path of hypernyms leading to a given synset, a model could also generate the synset's co-hyponyms or its hypernym's lexname 19 . our results suggest that hypernym prediction is challenging for words which occur less frequently in the corpus used to compute embeddings. methods are needed which can learn more effectively from low-frequency data, where the "unknown word" does not appear very often @xcite .