Low-resource Taxonomy Enrichment with Pretrained Language Models (2021)

Abstract: Taxonomies are symbolic representations of hierarchical relationships between terms or entities. While taxonomies are useful in broad applications, manually updating or maintaining them is labor-intensive and difficult to scale in practice. Conventional supervised methods for this enrichment task fail to find optimal parents of new terms in low-resource settings where only small taxonomies are available because of overfitting to hierarchical relationships in the taxonomies. To tackle the problem of low-resource taxonomy enrichment, we propose Musubu, an efficient framework for taxonomy enrichment in low-resource settings with pretrained language models (LMs) as knowledge bases to compensate for the shortage of information. Musubu leverages an LM-based classifier to determine whether or not inputted term pairs have hierarchical relationships. Musubu also utilizes Hearst patterns to generate queries to leverage implicit knowledge from the LM efficiently for more accurate prediction. We empirically demonstrate the effectiveness of our method in extensive experiments on taxonomies from both a SemEval task and real-world retailer datasets.

## introduction taxonomies, which represent the hierarchical relationships between terms, have been widely utilized in tasks related to information retrieval, recommendation, and classification @xcite @xcite . because the target domain of a specific taxonomy changes over time, taxonomies must be kept up to date so that newly introduced categories and hierarchical relationships can be properly integrated. however, manually constructing and maintaining taxonomies is a costly task due to their labor-intensive and domain-specific nature @xcite . the goal of a taxonomy enrichment task is to automate this costly maintenance @xcite , as shown in fig. 1 . taxonomy enrichment methods enrich a taxonomy that may be incomplete, i.e., a seed taxonomy, by predicting new hierarchical relationships between terms in the seed taxonomy and new terms. we focus on low-resource taxonomy enrichment tasks, in which there are few terms in the seed taxonomy, specifically less than 10,000. previous studies on hypernymy detection used a similar definition for low-resource settings @xcite . although conventional methods for taxonomy enrichment use large taxonomies (e.g., wordnet taxonomy (90,000 terms) @xcite , microsoft academic graph (355,000 terms) @xcite , most of the ones used in realistic situations (as seen in §4) have around a thousand terms, so they are categorized as low-resource settings. existing supervised methods @xcite @xcite train their parameters with term pairs in a seed taxonomy to find the optimal parent of a new term. these methods do not work in low-resource settings (as we verify in §4) because they require many term pairs. although @xcite partially tackled the problem of a small seed taxonomy, their proposed method requires additional information such as user search logs to improve the performance, which is not always available. to handle low-resource taxonomy enrichment tasks, we propose musubu, an efficient framework for such situations, with pretrained language models (lms) to compensate for the shortage of information on hierarchical relationships in a seed taxonomy. musubu leverages a pretrained lm-based classifier to determine whether term pairs have hierarchical relations for enriching taxonomies. the classifier can find the relationships accurately even if the training samples are limited because lms contain real-world term relationships in their parameters @xcite . musubu also utilizes hearst patterns @xcite for generating queries from term pairs to extract implicit knowledge related to them embedded in lms efficiently. we empirically demonstrate the effectiveness of our approach in the taxonomy enrichment task through extensive experiments on both the semeval-2015 task 17 @xcite taxonomies and real-world commerce taxonomies in amazon and walmart. our contributions are summarized as follows: ## problem statement in this section, we describe the notations and the problem definition of this paper. ## method the goal of low-resource taxonomy enrichment is to find the optimal parent @xmath0. we introduce a formulation of the taxonomy enrichment problem based on our probabilistic taxonomy model, then elaborate on our classification approach based on pretrained lms and hearst patterns. ## experiments in this section, we describe how we studied the performance of musubu on seven real-world tax- ## conclusion we proposed an efficient self-supervised approach, musubu, for low-resource taxonomy enrichment tasks. musubu utilizes a novel classifier based on pretrained lms and hearst patterns for generating queries. extensive experiments on taxonomy enrichment showed the effectiveness of musubu over the conventional approaches on the semeval and commerce taxonomies, especially in low-resource settings. taxoexpan @xcite . we used the authors' public implementation and set the hyperparameters by default in the original source code. to extract semantic features, we used fasttext as same as octet and msejrku. the number of trainable parameters in taxoexpan is 1.9m. octet @xcite . we extracted semantic and lexical features, input them into a twolayer feed-forward network with dropout layers, and used the output as the probability of terms. while the original method used graph-based features generated from user queries and the taxonomy in addition to the above features, we did not use them because we did not have any user queries. we constructed the lexical features following @xcite and the semantic features with fasttext @xcite trained by the common crawl dataset. to tune the model parameters, we used the same optimizer and the number of negative samples as that of our proposed method. the number of parameters in octet is 503k. msejrku @xcite . although this method was the winner of the semeval-2016 task 14, the implementation is not public. according to @xcite , the features used in octet are similar to those used in msejrku. we extracted the semantic and lexical features in octet, and then input a support vector machine.