B i RRE : Learning Bidirectional Residual Relation Embeddings for Supervised Hypernymy Detection (2020)

Abstract: The hypernymy detection task has been addressed under various frameworks. Previously, the design of unsupervised hypernymy scores has been extensively studied. In contrast, supervised classifiers, especially distributional models, leverage the global contexts of terms to make predictions, but are more likely to suffer from “lexical memorization”. In this work, we revisit supervised distributional models for hypernymy detection. Rather than taking embeddings of two terms as classification inputs, we introduce a representation learning framework named Bidirectional Residual Relation Embeddings (BiRRE). In this model, a term pair is represented by a BiRRE vector as features for hypernymy classification, which models the possibility of a term being mapped to another in the embedding space by hypernymy relations. A Latent Projection Model with Negative Regularization (LPMNR) is proposed to simulate how hypernyms and hyponyms are generated by neural language models, and to generate BiRRE vectors based on bidirectional residuals of projections. Experiments verify BiRRE outperforms strong baselines over various evaluation frameworks.

## introduction as a type of linguistic resources, hypernymy relations refer to "is-a" relations between terms. such relations are frequently exploited in a wide range of nlp tasks, including taxonomy induction @xcite , lexical entailment @xcite and web query understanding @xcite . in the nlp community, the task of hypernymy detection has been studied under various frameworks, e.g., unsupervised hypernym discovery @xcite @xcite , supervised hypernymy classification @xcite , graded lexical entailment @xcite . to address unsupervised hypernym discovery, pattern-based and distributional approaches are two mainstream types of methods. pattern-based approaches use hearst patterns @xcite and their variants to extract hypernymy relations from texts @xcite . distributional methods employ hypernymy measures (or called scores) to predict hypernymy based on distributional vectors @xcite , alleviating the pattern sparsity issue. @xcite combine hearst patterns and hyperbolic embeddings for unsupervised hypernym detection. compared to unsupervised tasks, the supervised hypernymy detection task is formulated more directly, classifying a term pair as hypernymy or non-hypernymy based on two terms' representations @xcite @xcite . although this task definition is more straightforward, the corresponding methods receive criticism because they may suffer from "lexical memorization" @xcite , referring to the phenomenon that they only learn whether a term is a "prototypical hypernym", rather than the actual relations between two terms. to address the problem, several methods combine other signals as inputs for hypernymy classifiers, such as dependency paths @xcite and the wordnet concept hierarchy @xcite . nonetheless, it is worth studying whether supervised classifiers can learn hypernymy relations purely based on distributional representations. in this paper, we revisit supervised distributional models for hypernymy detection, and propose a representation learning framework named bidirectional residual relation embeddings (birre). to handle "lexical memorization" @xcite , we learn a birre vector for each term pair as features for the classifier, avoiding using the two terms' embeddings directly. the birre vector models the possibility of a term being mapped to another in the embedding space by hypernymy relations, learned via existing neural language models and supervised signals of the training set. specifically, we introduce the latent projection model with negative regularization (lpmnr) to simulate how hypernyms and hyponyms are generated in the the embedding space. the birre vectors are generated based on bidirectional residuals of projection results of lpmnr. experiments over multiple public datasets and various evaluation frameworks prove that birre outperforms strong baselines. the rest of this paper is organized as follows. section 2 summarizes the related work. the birre framework is elaborated in section 3, with experiments shown in section 4. finally, we conclude our paper and discuss the future work in section 5. ## related work in this section, we overview related work on various tasks related to hypernymy detection. due to space limitation, we focus on recent advances and refer readers to @xcite for earlier work. pattern-based approaches date back to @xcite , utilizing handcrafted patterns in english for text matching. an example of hearst patterns is "[...] such as [...]". they are employed to build large-scale taxonomies @xcite . although hearst patterns are fairly simple, recent studies show they are highly useful for designing hypernymy measures @xcite . other approaches aim at improving the coverage of generalized hearst patterns by automatic pattern expansion @xcite , or considering other context-rich representations (such as heterogeneous information networks @xcite ). a potential drawback of pattern-based methods is that the recall of extraction results over specific domains is limited @xcite , as textual patterns are naturally sparse in the corpus. to overcome the sparsity issue, distributional hypernymy measures model the degree of hypernymy within a term pair. a majority of these hypernymy measures are based on distributional inclusion hypothesis (dih) @xcite , meaning that a hypernym covers a broader spectrum of contexts, compared to its hyponyms. the improvements and variants of dih include @xcite @xcite and many others. a comprehensive overview of distributional hypernymy measures can be found in @xcite . recently, @xcite combine hearst patterns and distributional vectors for hypernym detection. additionally, the work of graded lexical entailment @xcite and cross-lingual graded lexical entailment @xcite aims at computing numerical scores, indicating the degree of hypernymy of a term pair. for supervised hypernymy classification, traditional approaches employ distributional vectors of two terms as features, such as the concat model, the diff model, the simdiff model @xcite . recently, several approaches are proposed to learn hypernymy embeddings, considering the semantic hierarchies of concepts @xcite @xcite @xcite @xcite . for example, @xcite learn hypernym and hyponym embeddings for a term by max-margin neural network. @xcite propose hierarchical embeddings for hypernymy classification, jointly trained over texts and the wordnet concept hierarchy. @xcite propose a directional similarity neural network based on word embeddings to predict the degree of hypernymy between two terms. yet a number of models encode terms in the hyperbolic space, such as the hyperbolic lorentz model @xcite , hyperbolic entailment cones @xcite , and others @xcite . the hyperbolic geometry is more capable of modeling the transitivity property of hypernymy. additionally, patterns and distributional vectors can also be combined for supervised hypernymy prediction, as in @xcite ; @xcite and several systems submitted to semeval 2018 task 9 @xcite . another type of supervised models can be categorized as projection-based approaches, which model how to map embeddings of a term to those of its hypernyms. @xcite is most influential, followed by a number of variants. biemann et al. (2017); wang et al. (2017b, 2019b ) improve projection learning by considering explicit negative samples. the usage of orthogonal matrices is exploited in @xcite . one advantage is that they do not perform classification on two terms' embeddings directly, alleviating "lexical memoriza- ## the birre framework in this section, we first introduce the task description and the birre framework. the detailed steps and justifications are elaborated subsequently. ## experiments in this section, we conduct extensive experiments to evaluate the birre model over various benchmarks. we also compare it with state-of-the-art to show its effectiveness. ## conclusion and future work in this paper, we present the birre model for supervised hypernymy detection. it employs two projection-based hypernym and hyponym generation modules based on word embeddings to learn birre vectors for hypernymy classification. experimental results show that birre outperforms state-of-the-arts over various benchmark datasets. future work includes i) improving projection learning to model complicated linguistic properties of hypernymy; ii) extending our model to address other tasks, such as graded lexical entailment @xcite and cross-lingual graded lexical entailment @xcite ; and iii) exploring how deep neural language models (such as bert @xcite , transformer-xl @xcite , xlnet @xcite ) can improve the performance of hypernymy detection.