When Hearst Is not Enough: Improving Hypernymy Detection from Corpus with Distributional Models (2020)

Abstract: We address hypernymy detection, i.e., whether an is-a relationship exists between words (x ,y), with the help of large textual corpora. Most conventional approaches to this task have been categorized to be either pattern-based or distributional. Recent studies suggest that pattern-based ones are superior, if large-scale Hearst pairs are extracted and fed, with the sparsity of unseen (x ,y) pairs relieved. However, they become invalid in some specific sparsity cases, where x or y is not involved in any pattern. For the first time, this paper quantifies the non-negligible existence of those specific cases. We also demonstrate that distributional methods are ideal to make up for pattern-based ones in such cases. We devise a complementary framework, under which a pattern-based and a distributional model collaborate seamlessly in cases which they each prefer. On several benchmark datasets, our framework demonstrates improvements that are both competitive and explainable.

## introduction a taxonomy is a semantic hierarchy of words or concepts organized w.r.t. their hypernymy (a.k.a. is-a) relationships. being a well-structured resource of lexical knowledge, taxonomies are vital to various tasks such as question answering @xcite , textual entailment @xcite @xcite , and text generation @xcite . when automatically building taxonomies from scratch or populating manually crafted ones, the hypernymy detection task plays a central role. for a pair of queried words (x q , y q ), hypernymy detection requires inferring the existence of a hyponymhypernym relationship between x q and y q . due to the good coverage and availability, free-text corpora are widely used to facilitate hypernymy detection, resulting in two lines of approaches: patternbased and distributional. pattern-based approaches employ pattern pairs (x, y) extracted via hearst-like patterns @xcite , e.g., "y such as x" and "x and other y". an example of extracted pattern pairs from corpus are shown in figure 1 . despite their high precision, the extracted pairs suffer from sparsity which comes in two folds i.e., type-i: x q and y q separately appear in some extracted pairs, but the pair (x q , y q ) is absent e.g., (dog, animal); or type-ii: either x q or y q is not involved in any extracted pair e.g., @xcite . although matrix factorization @xcite or embedding techniques @xcite @xcite are widely adopted to implement pattern-based approaches, they only relieve the type-i sparsity and cannot generalize to unseen words appearing in the type-ii pairs. on the other hand, distribu-tional ones follow, or are inspired by, the distributional inclusion hypothesis (dih; @xcite , i.e., the set of the hyponym's contexts should be roughly contained by the hypernym's. although applicable to any word in a corpus, they are suggested to be inferior to pattern-based ones fed with sufficient extracted pairs @xcite . since pattern-based methods have unresolved sparsity issues, while distributional ones are more broadly applicable but globally inferior, neither of them can dominate the other in every aspect. in this light, we are interested in two questions: to answer the first question, we conduct analyses involving estimations on real-world corpora as well as statistics of common hypernymy detection datasets. results from both resources indicate that the likelihood of encountering the type-ii sparsity in practice could even reach up to more than 50%, which is thus non-negligible. for the second question, we present comhyper, a complementary framework (sec. 4.1) which takes advantage of both pattern-based models' superior performance on type-i cases and the broad coverage of distributional models on type-ii ones. specifically, to deal with type-ii sparsity, instead of directly using unsupervised distributional models, comhyper uses a training stage (sec. 4.3) to sample from output space of a pattern-based model to train another supervised distribution model implemented by different context encoders (sec. 4.2). in the inference stage, comhyper uses the two models to separately handle the type of sparsity they are good at, as illustrated in figure 1 . in this manner, comhyper relies on the partial use of pattern-based models on type-i sparsity to secure performance no lower than distributional ones, and further attempts to lift the performance by fixing the former's blind spots (type-ii sparsity) with the latter. on several benchmarks and evaluation settings, the distributional model in comhyper proves effective on its targeted cases, making our complementary approach outperform a competitive class of pattern-based baselines @xcite . further analysis also suggests that comhyper is robust when facing different mixtures of type-i and -ii sparsity. our contributions are summarized as : 1) we confirm that a specific type of sparsity issue of current pattern-based approaches is non-negligible. 2) we propose a framework of complementing patternbased approaches with distributional models where the former is invalid. 3) we systematically conduct comparisons on several common datasets, validating the superiority of our framework. to infer unknown hypernymies between known words, e.g., implicit is-a pairs in transitive closures, pattern-based models are proposed. @xcite and @xcite show that, on a broad range of benchmarks, simple matrix decomposition or embeddings on pattern-based word cooccurrence statistics provide robust performance. on probase @xcite -a hearst-patternbased taxonomy, @xcite use embeddings to address the same sparsity problem. some methods @xcite @xcite @xcite embed wordnet in low-dimensional space. depending on vectors of words learnt from known is-a pairs, the above pattern-based methods cannot induce more hypernymy pairs whose words do not appear in any pattern. distributional approaches. distributional models are inspired by dih @xcite . they work on only word contexts rather than extracted pairs, thus are applicable to any word in a corpus. early unsupervised models typically propose asymmetric similarity metrics over manual word feature vectors for entailment @xcite @xcite . @xcite and @xcite , the authors inject dih into unsupervised embedding models to yield latent feature vectors with hypernymy information. those feature vectors, manual or latent, may serve in unsupervised asymmetric metrics or to train supervised hypernymy classifiers. @xcite explore combinations of manual features and (un)supervised predictors, and suggest that unsupervised metrics are more robust w.r.t. the distribution change of training instances. projection learning @xcite @xcite has been used for supervised hypernymy detection. other improved methods. due to weak generalization ability of hearst patterns, @xcite and @xcite relieve the constraints from strict hearst patterns to co-occurring contexts or lexico-syntactic paths between two words. they encode the co-occurring contexts or paths using word vectors to train hypernymy embeddings or classifiers. although leading to better recall than hearst patterns @xcite , they limit the trained embeddings or models from generalizing to every word in a corpus. nevertheless they have no ability to cope with the type-ii sparsity, which is the main focus of our work. another line of retrofitting methods @xcite , i.e., adjusting distributional vectors to satisfy external linguistic constraints, has been applied to hypernymy detection. however, they strictly require more additional resources e.g., synonym and antonym to achieve better performance @xcite . to the best of our knowledge, we are the first to propose complementing the two lines of approaches to cover every word in a simple yet efficient way, with extensive analysis of the framework's potential and evaluation of performances. ## preliminaries we formally define the aforementioned two types of sparsity, and provide some statistical insights about their impacts on pattern-based methods. ## experimental setup we adopt the widely-used comprehensive evaluation framework foot_0 provided by @xcite ; @xcite . to make experimental results comparable, we align the settings as much as possible. ## experimental results we aim to answer: 1) are our distributional models supervised well by the pattern-based model? 2) do they improve our complementary methods over the pattern-based ones? 3) are complementary methods robust w.r.t. fewer extracted pairs? ## conclusion and future work we propose complementing pattern-based and distributional methods for hypernymy detection. as far as we know, this is the first work along this line. we formally depict two types of sparsity that extracted pairs face, and indicate that patternbased methods are invalid on the type-ii, i.e., outof-pattern pairs. by analyzing common corpora and datasets, we confirm that oop pairs are nonnegligible for the task. to this end, we devise a complementary framework, where a pattern-based and distributional model handle ip and oop pairs separately, while collaborating seamlessly to give unified scores. oracle performance analysis shows that our framework has high potentials on several datasets. supervised by the pattern-based model, the distributional model shows robust capability of scoring oop pairs and pushing the overall performance towards the oracle bounds. in the future, we will extend the similar approach to multilingual @xcite or crosslingual @xcite ) lexical entailment tasks. moreover, one interesting direction is to use hyperbolic embeddings @xcite for pattern-based models due to their inherent modeling ability of hierarchies.