Find Parent then Label Children: A Two-stage Taxonomy Completion Method with Pre-trained Language Model (2023)

Abstract: Taxonomies, which organize domain concepts into hierarchical structures, are crucial for building knowledge systems and downstream applications. As domain knowledge evolves, taxonomies need to be continuously updated to include new concepts. Previous approaches have mainly focused on adding concepts to the leaf nodes of the existing hierarchical tree, which does not fully utilize the taxonomy’s knowledge and is unable to update the original taxonomy structure (usually involving non-leaf nodes). In this paper, we propose a two-stage method called ATTEMPT for taxonomy completion. Our method inserts new concepts into the correct position by finding a parent node and labeling child nodes. Specifically, by combining local nodes with prompts to generate natural sentences, we take advantage of pre-trained language models for hypernym/hyponymy recognition. Experimental results on two public datasets (including six domains) show that ATTEMPT performs best on both taxonomy completion and extension tasks, surpassing existing methods.

## introduction taxonomies foot_0 are an important form of domain knowledge that organize concepts into hierarchical structures, representing "hypernym-hyponym" relationships among concepts in the form of trees or directed acyclic graphs @xcite . taxonomies are essential components of knowledge systems such as ontologies and knowledge graph @xcite , and are widely used in various downstream applications, including search engineering @xcite , recommendation systems @xcite , figure 1 : an example of taxonomy completion for a non-leaf node. the new concept "social science" needs to be inserted into the correct position in the existing taxonomy. as domain knowledge continues to evolve, especially with the rapid growth of web content, new concepts are constantly emerging. in order to stay current, original taxonomies must incorporate these new concepts and adapt their hierarchical relationships. for example, as shown in figure 1 , with the advancement of sociology and science, the concept of "social science" should be added to the science knowledge system, and the original structure should be adjusted accordingly. however, existing taxonomies are primarily constructed by human experts @xcite . manual extraction of domain concepts and detection of hierarchical relationships by domain experts is both time-consuming and labor-intensive, and may result in missing important concepts and relationships. to extend existing taxonomies automatically, researchers have proposed the tasks of taxonomy expansion (te) and taxonomy completion (tc). both tasks aim to append new nodes (concepts) to a given taxonomy . the main difference is that te focuses on identifying the parent of a given node (usually a leaf node), while tc aims to identify both the parent and child nodes. as illustrated in figure 1 , te would aim to identify the parent node of "social science", while tc would also aim to identify the child nodes of "social science". recently, researchers have been focusing on using pre-trained language models, such as bert @xcite , to improve the performance of taxonomy expansion @xcite . for example, temp @xcite appends new concepts to leaf nodes and generates candidate taxonomy paths, then uses a pre-trained model for ranking and selecting the best path. musubu @xcite generates candidate "hypernym"-"new concept" pairs using hearst patterns @xcite , and relies on pre-trained knowledge to identify the optimal hypernym node for the new concept. these proposed models have greatly improved the effectiveness of taxonomy updates, thanks to the improved generalization performance of pre-trained language models @xcite . although current te&tc methods have achieved good results, there are several main issues that need to be addressed. firstly, existing te methods struggle to extend non-leaf nodes or perform poorly in this task @xcite . secondly, while existing tc methods can extend both leaf and non-leaf nodes, they may be less effective in leaf node expansion than specialized te methods @xcite , potentially due to a lack of sufficient utilization of knowledge. furthermore, these methods often require large amounts of labeled samples or external resources, which are not always available @xcite . lastly, current tc methods do not typically involve modifying the nodes of the original taxonomy system (all original parent-child relationships are preserved after adding nodes to the taxonomy). however, the insertion of new nodes can modify the relationship of the original nodes. for example, the insertion of "social science" in figure 1 would change the relationship between "science-anthropology" from father-son to grandfather-grandson. to address these issues, we propose a twostage taxonomy completion method with pre-trained language model (attempt), which inserts new concepts into the correct position by identifying a parent node and labeling child nodes. in the first stage of our proposed method, we use the "taxonomy-path prompt with pretrained model" (ppt) approach to take advantage of the local information of the taxonomy path and convert it into natural language using a prompt method, which helps to better utilize the implicit knowledge of the pre-trained model. additionally, the pre-trained model's extensive knowledge reserve allows us to avoid the need for external resources and large amounts of labeled data. in the second stage, we propose the "multiple nodes labeling" (mnl) method, which jointly identifies each child node and better utilizes the interdependence between nodes, resulting in more accurate node type prediction (including fatherson, sibling and other relationships). additionally, mnl allows for modification of the original taxonomy nodes and simultaneous annotation of multiple child nodes. we conduct detailed experiments on two public datasets (including six domains) to evaluate the effectiveness of our proposed method, attempt, in leaf and non-leaf node expansion. specifically, for leaf nodes, our parent-finding method (ppt) outperforms the best baseline by 8.2% in accuracy. for non-leaf nodes, our children-finding method (mnl) improves by 21% and 20.3% respectively in accuracy and average f1 score, compared to a pair-wise classification method. on the overall task, our proposed method (attempt) outperforms other methods by 2.4% in average f1 score. in summary, the main contributions of this paper include: ## related work taxonomy construction aims to build a treestructured taxonomy with a set of terms from scratch. existing methods can be roughly divided into two categories. the first is an unsupervised method to construct the taxonomy based on clustering @xcite @xcite . the terms are grouped into a hierarchy based on hierarchical clustering or topic models @xcite . each node of this taxonomy is a collection of topicindicative terms, different from the taxonomy in this paper (each node represented by one individual term). the other approach constructs a taxonomy based on terms, where each node represents a term concept @xcite . hypernymy detection models are often used for this task. for example, pattern-based @xcite @xcite or distributional models @xcite @xcite extract the hypernymy for a given query node and then organize them into a tree structure. creating a taxonomy from scratch is laborintensive. in many scenarios, such as e-commerce, some taxonomies may already be deployed in online systems, which involves the demand of taxonomy extension. qassit @xcite is a semi-supervised vocabulary classification method, mainly based on genetic algorithms. the taxi @xcite system uses a taxonomy induction method based on lexicosyntactic patterns, substrings, and focused crawling. later, taxogen @xcite uses term embeddings and hierarchical clustering to construct topic taxonomies recursively. temp @xcite is a self-supervised classification extension method that trains models with a new dynamic margin loss margin function. taxonomy completion @xcite is a recently proposed task that aims to find appropriate hypernyms-hyponyms for new nodes, not just hypernyms. gentaxo @xcite gathers information from complex local structural information and learns to generate full names of concepts from corpora. tmn @xcite focuses on channel gating mechanisms and triplet matching networks. corel relies on concept learning and relation transferring to build a seedoriented topic taxonomy. but the above mentioned methods also have some issues. the addition of new nodes may also lead to changes in the original taxonomy. the taxonomy completion task only finds the hyponyms of a given node, which cannot modify of the original taxonomy. gentaxo @xcite requires a large amount of training data to learn enough information, and corel @xcite focuses more on topic taxonomy than the taxonomy of individual terms. other works such as cgexpan @xcite use the automatically generated class names and the classguided entity selection module for entity expansion. however, cgexpan @xcite ) is more on the entity set than the tree taxonomy. in addition, although the above methods can find both hypernyms and hyponyms of a given query node, they do not make sufficient use of the pre-trained model or do not use the pre-trained model at all @xcite . this may lead them to perform poorly on the hypernym recognition task, inferior to the specialized taxonomy extension methods of the pretrained model @xcite . and most methods of taxonomy extension cannot perform well on the task of taxonomy completion @xcite . we are dedicated to finding an approach that works in both tasks. ## method given an existing taxonomy @xmath0, e) and a set of new terms v ′ , where v is a set of terms, and e is a set of " hyponym-hypernym" relationships between terms, the task of taxonomy completion is to insert the new terms v ′ ∈ v ′ into the appropriate position of the existing taxonomy t one by one and extend them into a more complete taxonomy @xmath1, ẽ). figure 2 provided illustrates the overall structure of the attempt method, which is broken down into two main stages: the parent finding stage and the children finding stage. these two stages work together to identify the relationships between terms in the taxonomy, specifically determining the parent and children of a given term. ## experiments in this section, we first describe the experimental setup and implementation details in section 4.1 and section 4.2. we then present the results of our experiments in section 4.3, including a comparison of our approach to the baseline method. to further understand the contribution of different components of our approach, we conduct ablation experiments in section 4.4 to investigate the effectiveness of using local information and prompts in attempt. ## conclusion this paper proposes a two-stage taxonomy completion method based on pre-trained language models (attempt), which effectively inserts the new concept in the correct position by finding a parent node then labeling children nodes. in addition, we use prompt to generate natural language information suitable for the pre-trained model further to improve the effectiveness of parent node recognition and children labeling for the given node. our experiments on two types and three domains with six datasets show that our method can enhance the effectiveness of locating the position of a given node in existing taxonomies. furthermore, the efficacy of local information and prompts in attempt is also demonstrated by ablation experiments. in conclusion, our proposed attempt method is an effective approach for taxonomy completion, and it can be further improved with more comprehensive datasets.