T aco P rompt: A Collaborative Multi-Task Prompt Learning Method for Self-Supervised Taxonomy Completion (2023)

Abstract: Automatic taxonomy completion aims to attach the emerging concept to an appropriate pair of hypernym and hyponym in the existing taxonomy. Existing methods suffer from the overfitting to leaf-only problem caused by imbalanced leaf and non-leaf samples when training the newly initialized classification head. Besides, they only leverage subtasks, namely attaching the concept to its hypernym or hyponym, as auxiliary supervision for representation learning yet neglect the effects of subtask results on the final prediction. To address the aforementioned limitations, we propose TacoPrompt, a Collaborative Multi-Task Prompt Learning Method for Self-Supervised Taxonomy Completion. First, we perform triplet semantic matching using the prompt learning paradigm to effectively learn non-leaf attachment ability from imbalanced training samples. Second, we design the result context to relate the final prediction to the subtask results by a contextual approach, enhancing prompt-based multi-task learning. Third, we leverage a two-stage retrieval and re-ranking approach to improve the inference efficiency. Experimental results on three datasets show that TacoPrompt achieves state-of-the-art taxonomy completion performance. Codes are available at https://github.com/cyclexu/TacoPrompt.

## introduction a taxonomy is a tree-like hierarchical structure organized around hypernym-hyponym ("is-a") relations between concepts. it has been widely used in downstream natural language processing (nlp) tasks such as named entity recognition @xcite , transfer learning @xcite and language models @xcite . most of the existing taxonomies are maintained by domain experts manually, which is labourintensive and time-consuming. as new concepts figure 1 : an example of attaching new concepts to the existing "computer science" taxonomy by triplet semantic matching using prompt learning paradigm. several recent studies have achieved promising progress in taxonomy completion. typically, they first leverage semantic @xcite or structural @xcite information to generate the concept and the position representation using multiple tasks as supervision @xcite . then, these methods utilize the matching module to identify the attachment prediction for the given query based on the generated representation. although achieving remarkable performance, these representation-based methods have two main limitations. first, they primarily leverage a newly initialized classification head as the matching module, whose ability overfits to leafonly, i.e., the taxonomy completion task degrades to the expansion task @xcite , when learning from imbalanced leaf and non-leaf training samples. second, these methods only leverage subtasks, namely attaching the concept to its hypernym or hyponym, as auxiliary supervision for representation learning @xcite @xcite while neglecting the results of subtasks, which are proven effective in the final prediction @xcite . to effectively address the aforementioned limitations, we propose tacoprompt for self-supervised taxonomy completion in this paper. first, we leverage the pre-trained language model (plm) by the prompt learning paradigm to perform triplet semantic matching for the hierarchical "is-a" relations prediction between the query, candidate hypernym and hyponym concepts, as illustrated in figure 1 . prompt learning paradigm @xcite reuses the plm's masked language modelling (mlm) head instead of introducing the randomly initialized classification head, requiring fewer training samples to learn the target ability. thus, we leverage it to address the overfitting problem caused by imbalanced training samples. second, we extend the multi-task learning from representation-based methods to the crossencoder-based framework by taking advantage of prompt learning's effectiveness in multi-task learning scenarios @xcite . moreover, we design the result context to relate the final prediction to the subtask results by a contextual approach to enhance prompt-based multi-task learning. specifically, we design the hard result context using explicit answer tokens and the soft result context using learnable hidden vectors. third, we leverage a two-stage retrieval and re-ranking approach to reduce the cross-encoder's expensive inference computation costs, increasing the inference efficiency by up to 1,000 times. we conduct extensive experiments on three taxonomy datasets to compare tacoprompt with stateof-the-art taxonomy expansion and completion methods. the comparative results show that taco-prompt outperforms previous state-of-the-art methods by a large margin, such as by 11.1% in hit@1 and 9.1% in recall@10 on average. our contributions can be summarized as follows: ## methodology in this section, we first introduce our triplet semantic matching pipeline using the prompt learning for taxonomy completion in §4.1. then, we enhance it by collaborative multi-task learning via the contextual approach in §4.2. furthermore, we discuss the efficient inference process based on a two-stage approach in §4.3 and describe the self-supervised training details in §4.4. ## experiments detailed experimental settings, including the introduction of datasets, evaluation metrics and baselines, are described in appendix a. ## conclusion in this paper, we propose tacoprompt for selfsupervised taxonomy completion. specifically, we leverage the prompt learning paradigm to perform triplet semantic matching, addressing the overfitting to the leaf-only problem caused by imbalanced training samples. then, we perform collaborative multi-task learning by designing the result context to enhance the prompt learning framework. moreover, we propose a two-stage retrieval and re-ranking approach for efficient inference. experimental results on three taxonomy datasets show that tacoprompt outperforms the state-of-the-art methods by a large margin.