H yper E xpan: Taxonomy Expansion with Hyperbolic Representation Learning (2021)

Abstract: Taxonomies are valuable resources for many applications, but the limited coverage due to the expensive manual curation process hinders their general applicability. Prior works attempt to automatically expand existing taxonomies to improve their coverage by learning concept embeddings in Euclidean space, while taxonomies, inherently hierarchical, more naturally align with the geometric properties of a hyperbolic space. In this paper, we present HyperExpan, a taxonomy expansion algorithm that seeks to preserve the structure of a taxonomy in a more expressive hyperbolic embedding space and learn to represent concepts and their relations with a Hyperbolic Graph Neural Network (HGNN). Specifically, HyperExpan leverages position embeddings to exploit the structure of the existing taxonomies, and characterizes the concept profile information to support the inference on new concepts that are unseen during training. Experiments show that our proposed HyperExpan outperforms baseline models with representation learning in a Euclidean feature space and achieves state-of-the-art performance on the taxonomy expansion benchmarks.

## introduction taxonomy, a systematic categorization scheme, is an effective way to organize and classify knowledge @xcite . taxonomies have been used to support many downstream applications such as content management in e-commerce @xcite , web search @xcite , digital libraries @xcite , and nlp tasks @xcite @xcite . the curation of taxonomies mostly relies on human experts, which can be timeconsuming and expensive, and hence suffer from limited coverage of the knowledge (jurgens and * equal contributions. pilehvar, 2016). to alleviate this issue and handle constantly emerging new concepts, automating the taxonomy construction has attracted attentions from the research community @xcite . one type of such automated taxonomy curation is taxonomy expansion, which enriches an existing taxonomy to incorporate new and broader concepts. specifically, the expansion of a taxonomy is performed as attaching new concept nodes to proper positions of a seed taxonomy graph, which is usually represented as a hierarchical tree @xcite . to systematically enrich a taxonomy graph, concept embeddings are firstly learned by structurally characterizing the concepts in the existing taxonomies, which are then used to match the embeddings of query concepts for the expansion. prior works learn the concept embeddings with local structural features, such as edge semantic representation @xcite and graph neural networks (gnn) @xcite . however, as a concept can lead to multiple subconcepts, the sizes of taxonomies expand exponentially with respect to their levels. the euclidean embedding space, where existing works commonly build upon, fails to account for this property. in contrast, a hyperbolic space @xcite , where the circumference of a negative-curved space grows exponentially with regard to the radius as illustrated in figure 1 , can better capture such special characteristics of taxonomies. in this paper, we present hyperexpan, a taxonomy expansion framework based on hyperbolic representation learning, that: (1) better preserves the taxonomical structure in a more expressive hyperbolic space, (2) effectively characterizes concepts by exploiting sparse neighborhood information beyond standard parent-child relations @xcite , and (3) improves inference precision and generalizability by leveraging pretrained distributional features. 1 specifically, hyperexpan incorporates two types of features to exploit the structural presentation of a taxonomy: a relative positional embedding of a node depending on its relation to the anchor node, and an absolute positional embedding defined by its depth within a taxonomy. hyper-expan first constructs an ego subgraph around the potential attaching candidate concepts, i.e. the anchor concepts, and then leverages a hyperbolic graph neural network (hgnn) to obtain the anchor concept embeddings. a parent-child matching score for the attachment is subsequently produced by comparing both the anchor and query concept embeddings in the same hyperbolic space. we evaluate hyperexpan on wordnet and microsoft academic graph datasets. experiments show that the learned hyperbolic concept embeddings achieve better expansion performance than the euclidean counterpart, outperforming the stateof-the-art models. we also perform ablation studies to demonstrate the effectiveness of each component and the design choice of hyperexpan. our contributions are summarized as follows: (1) we present an effective and generalizable taxonomy expansion framework via hyperbolic representation learning. (2) we introduce methods to incorporate pretrained distributional features and taxonomyspecific information in the hyperbolic gnn design. ## preliminaries we introduce preliminaries about hyperbolic geometry and then define the task. 1 code is available at https://github.com/ pluslabnlp/hyperexpan ## hyperexpan we propose hyperexpan, a taxonomy expansion framework based on hyperbolic geometry and gnns. as shown in figure 2 , hyperexpan consists of the following main steps: 1) initial concept feature generations utilizing the profile information ( § 3.1). 2) encoding query and anchor concept features with hyperbolic (graph) neural networks ( § 3.2). 3) computing the query-anchor embedding matching scores for attaching query concepts to proper anchor positions ( § 3.3). we will describe each step in details and how to train the matching model ( § 3.4) in the following sections. ## experiments we evaluate the hyperexpan and its variants on four large-scale real-world taxonomies utilized by @xcite and @xcite . ## conclusion and future work we present hyperexpan, a taxonomy expansion model which better preserves the taxonomical structure in an expressive hyperbolic space. we use an hgnn to incorporate neighborhood information and positional features of concepts, as well as profile features that are essential to jump-start zero-shot concept representations. experimental results on wordnet and microsoft academic graph taxonomies show that hyperexpan performs better than its euclidean counterparts and consistently outperforms state-of-the-art taxonomy expansion models. in the future, we plan to extend hyperex-pan for inducing dynamic taxonomies @xcite and taxonomy alignment @xcite .