TEMP : Taxonomy Expansion with Dynamic Margin Loss through Taxonomy-Paths (2021)

Abstract: As an essential form of knowledge representation, taxonomies are widely used in various downstream natural language processing tasks. However, with the continuously rising of new concepts, many existing taxonomies are unable to maintain coverage by manual expansion. In this paper, we propose TEMP, a self-supervised taxonomy expansion method, which predicts the position of new concepts by ranking the generated taxonomy-paths. For the first time, TEMP employs pre-trained contextual encoders in taxonomy construction and hypernym detection problems. Experiments prove that pre-trained contextual embeddings are able to capture hypernym-hyponym relations. To learn more detailed differences between taxonomy-paths, we train the model with dynamic margin loss by a novel dynamic margin function. Extensive evaluations exhibit that TEMP outperforms prior state-of-the-art taxonomy expansion approaches by 14.3% in accuracy and 15.8% in mean reciprocal rank on three public benchmarks.

## introduction taxonomies, tree-structured semantic hierarchies that organize entities by hypernym-hyponym (is-a) relations, play an important role in many nlp tasks such as question answering @xcite , query understanding @xcite and information extraction @xcite . manually curated taxonomies usually face the limited coverage issue, especially when new concepts arise continuously. a low coverage taxonomy can largely hurt the performance of downstream tasks relied on it. moreover, for maintaining and expanding existing taxonomies, the curation process that requires domain experts is expensive and time-consuming. thus, we study the automatic taxonomy expansion task (figure 1 ): given an existing taxonomy, a text corpus, and a set of concepts, the * corresponding author goal is to expand the taxonomy by inserting concepts into it. two common strategies used to study taxonomy construction and expansion are pattern-based methods (e.g. the hearst pattern @xcite ) and distributional methods @xcite . recent evidence suggests that the semantic information or structural features encoding in the representation is an effective way to solve the task, especially probability statistics from a large corpus @xcite , semantic information extracted from text data @xcite , and properties of hypernym-hyponym relations such as strict partial order @xcite . since taxonomies can be formulated as directed acyclic graphs (dags), the graph structure has been seen as important information for taxonomy expansion and construction in recent works @xcite . however, according to our observation, the path composed of ancestor nodes in a taxonomy is a more appropriately encoded object in hypernym-hyponym relations. in a tree-structured taxonomy, all the ancestor nodes have an is-a relation with the child node. in figure 1 , for example, "science" -"systematics" -"biosystematics" is the taxonomy-path of word "biosystematics". "biosystematics" not only "is-a" "systematics" but also "is-a" "science". in addition, the serial structure of taxonomy-path is also more appropriate than the graph structure for transformers to encode. as far as we know, there has been no attempt to take pre-trained contextual encoders (such as bert @xcite ) as the core of taxonomy expansion or construction model. pre-trained contextual encoders have been proved powerful in various nlp tasks such as question answer(qa) @xcite , information retrieval (ir) @xcite in linguistics, deixis is the use of general words and phrases to refer .. ## related work automatic taxonomy construction has been a longterm task in literature in the last few decades. most existing methods follow the paradigm of constructing taxonomy from scratch. they firstly @xmath0, @xmath1. these approaches exploit semantic information and structural features such as lexical-patterns @xcite or distributional embeddings @xcite @xcite to automatically construct taxonomies. however, recent practical applications have revealed that it is laborious to construct taxonomies from scratch when facing the continuously rising of new concepts, so solutions to the taxonomy expansion task are in urgent need. in linguistics, deixis is the use of general words and phrases to refer ... deixis sociolinguistics linguistics science deixis semantics linguistics science encoder encoder margin loss science linguistics semantics deixis sociolinguistics systematics existing taxonomy concept definition positive taxonomy path calculate margin negative taxonomy path recently, numerous methods have been proposed to solve the aforementioned problem @xcite @xcite @xcite . for example, @xcite proposes a position-enhanced graph neural network framework to encode the local structure of an anchor concept with a noise-robust training objective. @xcite converts candidate anchor positions from the whole existing taxonomy to mini-paths, in which way can better capture and integrate multiple sources of information via a multi-view co-training procedure. @xcite first designs a realistic approach to demonstratively model unobserved and heterogeneous edge semantics. @xcite generalizes expansion task to the more general "one-to-pair" completion task and applies primal and auxiliary scorers based on the neural tensor network to rank candidate anchor positions. as far as we know, all the existing methods attempt to determine the attachment position by scoring between several nodes, we are the first to take the path as the unit for encoding and calculating scores. besides, to the best of our knowledge, few state-of-the-art expansion approaches encode information out of supervision information in the existing taxonomy, we take pre-trained contextual encoder as core to aggregate more valuable information and resources such as word definition to improve performance. ## the temp method in this section, we describe our proposed method temp. first, we introduce taxonomy-path, an im-portant concept in our method(section 3.1). our model takes concept definitions and taxonomypaths as input and relies on the pre-trained contextual encoders as its core (section 3.2). the parameters of the model are trained by margin ranking loss (mrl) with a dynamic margin function designed for taxonomy expansion (section 3.3). finally, we discuss how to sample self-supervision data and fine-tune the model with dynamic margin loss (section 3.4). ## experiments in this section, we first introduce the experimental setup (section 4.1) and report the overall performance compared with baselines (section 4.2). then, we study the effectiveness of the key choices in temp by ablation experiments (section 4.3). furthermore, we discuss the factors that can affect the performance of temp (section 4.4). datasets. we evaluate temp using all the three english datasets in semeval-2016 task 13 2 @xcite . these datasets correspond to human-curated concept taxonomies of three different domains: environment, science, food (summarized in ## conclusion we proposed temp, a self-supervised method for taxonomy expansion, that relies on the pre-trained contextual encoder as its core. temp takes the definition of the query concept and the generated taxonomy-path as input to predict the attachment position. the model is trained by a margin ranking loss with a novel dynamic margin function to better capture the semantic similarity between taxonomypaths. experiments on three datasets from different domains show that temp outperforms state-of-theart methods. further ablation studies show that our key choices in temp have an effect on the performance in varying degrees especially the use of margin loss. for future work, we plan to design sampling methods for temp to improve its performance and robustness. we also want to do interpretability studies about the effect of margin loss in model training.