Taxonomy Construction of Unseen Domains via Graph-based Cross-Domain Knowledge Transfer (2020)

Abstract: Extracting lexico-semantic relations as graph-structured taxonomies, also known as taxonomy construction, has been beneficial in a variety of NLP applications. Recently Graph Neural Network (GNN) has shown to be powerful in successfully tackling many tasks. However, there has been no attempt to exploit GNN to create taxonomies. In this paper, we propose Graph2Taxo, a GNN-based cross-domain transfer framework for the taxonomy construction task. Our main contribution is to learn the latent features of taxonomy construction from existing domains to guide the structure learning of an unseen domain. We also propose a novel method of directed acyclic graph (DAG) generation for taxonomy construction. Specifically, our proposed Graph2Taxo uses a noisy graph constructed from automatically extracted noisy hyponym hypernym candidate pairs, and a set of taxonomies for some known domains for training. The learned model is then used to generate taxonomy for a new unknown domain given a set of terms for that domain. Experiments on benchmark datasets from science and environment domains show that our approach attains significant improvements correspondingly over the state of the art.

## introduction taxonomy has been exploited in many natural language processing (nlp) applications, such as question answering @xcite , query understanding @xcite , recommendation systems @xcite , etc. automatic taxonomy construction is highly challenging as it involves the ability to recognize -(i) a set of types (i.e. hypernyms) from a text corpus, (ii) instances (i.e. hyponyms) of each type, and (iii) is-a (i.e. hypernymy) hierarchy between types. existing taxonomies (e.g., wordnet @xcite ) are far from being complete. tax-onomies specific to many domains are either entirely absent or missing. in this paper, we focus on construction of taxonomies for such unseen domains foot_0 . since taxonomies are expressed as directed acyclic graphs (dags) @xcite , taxonomy construction can be formulated as a dag generation problem. there has been considerable research on graph neural networks (gnn) @xcite over the years; particularly inspired by the convolutional gnn @xcite where graph convolution operations were defined in the fourier domain. in a similar spirit to convolutional neural networks (cnns), gnn methods aggregate neighboring information based on the connectivity of the graph to create node embeddings. gnn has been applied successfully in many tasks such as matrix completion @xcite , manifold analysis @xcite , predictions of community @xcite , knowledge graph completion @xcite , and representations of network nodes @xcite . to the best of our knowledge, there has been no attempt to exploit gnn for taxonomy construction. our proposed framework, graph2taxo, is the first to show that a gnn-based model using a crossdomain noisy graph can substantially improve the taxonomy construction of unseen domains (e.g., environment) by exploiting taxonomy of one or more seen domains (e.g., food). (the task is described in detail in section 3.1.) another novelty of our approach is we are the first to apply the acyclicity constraint-based dag structure learning model @xcite for taxonomy generation task. the input of graph2taxo is a cross-domain noisy graph constructed by connecting noisy candidate is-a pairs, which are extracted from a large corpus using standard linguistic pattern-based approaches @xcite . it is noisy because pattern-based approaches are prone to poor coverage as well as wrong extractions. in addition, it is cross-domain because the noisy is-a pairs are extracted from a large-scale corpus which contains a collection of text from multiple domains. our proposed neural model directly encodes the structural information from a noisy graph into the embedding space. since the links between domains are also used in our model, it has not only structural information of multiple domains but also crossdomain information. we demonstrate effectiveness of our proposed method on science and environment datasets @xcite , and show significant improvements on f-score over the state of the art. ## related work taxonomy construction (also known as taxonomy induction) is a well-studied problem. most of the existing works follow two sequential steps to construct taxonomies from text corpora @xcite . first, is-a pairs are extracted using patternbased or distributional methods. then, a taxonomy is constructed from these is-a pairs. the pattern-based methods, pioneered by @xcite , detect is-a relation of a term pair (x, y) using the appearance of x and y in the same sentence through some lexical patterns or linguistic rules @xcite . @xcite represented each (x, y) term-pair as the multiset of dependency paths connecting their co-occurrences in a corpus, which is also regarded as a path-based method. an alternative approach for detecting is-a relation is the distributional methods @xcite , using the distributional representation of terms to directly predict relations. as for the step of taxonomy construction using the extracted is-a pairs, most of the approaches do it by incrementally attaching new terms @xcite @xcite . @xcite is the first to present a reinforcement learning based approach, named taxorl, for this task. for each term pair, its representation in taxorl is obtained by the path lstm encoder, the word embeddings of both terms, and the embeddings of features. recently, @xcite argued that strict partial orders foot_1 correspond more directly to dags. they proposed a neural network architecture, called strict partial order network (spon), that enforces asymmetry and transitive properties as soft constraints. empirically, they showed that such a network produces better results for detecting hyponym-hypernym pairs on a number of datasets for different languages and domains in both supervised and unsupervised settings. many graph-based methods such as @xcite and @xcite regard the task of hypernymy organization as a hypernymy detection problem followed by a graph pruning problem. for the graph pruning task, various graphtheoretic approaches such as optimal branching algorithm @xcite , edmond's algorithm @xcite ) and tarjan's algorithm @xcite have been used over the years. in addition to these, @xcite mentions several other graphbased taxonomy induction approaches. in contrast, our approach formulates the taxonomy construction task as a dag generation problem instead of an incremental taxonomy learning @xcite , which differentiates it when compared with the existing methods. in addition, our approach uses the knowledge from existing domains @xcite to build the taxonomies of missing domains. ## the graph2taxo framework in this section, we first formulate the problem statement and then introduce our proposed graph2taxo framework as a solution. we describe the individual components of this framework in detail, along with justifications of how and why these components come together as a solution. ## experiments we evaluate graph2taxo on semeval-2016 task 13: taxonomy extraction evaluation 3 , otherwise known as texeval-2 task @xcite . all experiments are implemented in pytorch. code is publicly available at https://github.com/ibm/ gnn-taxo-construction . ## conclusion we have introduced a gnn-based cross-domain knowledge transfer framework graph2taxo, which makes use of a cross-domain graph structure, in conjunction with an acyclicity constraint-based dag learning for taxonomy construction. furthermore, our proposed model encodes acyclicity as a soft constraint and shows that the overall model outperforms state of the art. in the future, we would like to figure out different strategies to merge individual gains, obtained by separate application of the dag constraint, into a setup that can take the best of both precision and recall improvements, and put forth a better performing system. we also plan on looking into strategies to improve recall of the constructed taxonomy.