Constructing Taxonomies from Pretrained Language Models (2021)

Abstract: We present a method for constructing taxonomic trees (e.g., WordNet) using pretrained language models. Our approach is composed of two modules, one that predicts parenthood relations and another that reconciles those pairwise predictions into trees. The parenthood prediction module produces likelihood scores for each potential parent-child pair, creating a graph of parent-child relation scores. The tree reconciliation module treats the task as a graph optimization problem and outputs the maximum spanning tree of this graph. We train our model on subtrees sampled from WordNet, and test on nonoverlapping WordNet subtrees. We show that incorporating web-retrieved glosses can further improve performance. On the task of constructing subtrees of English WordNet, the model achieves 66.7 ancestor F1, a 20.0% relative increase over the previous best published result on this task. In addition, we convert the original English dataset into nine other languages using Open Multilingual WordNet and extend our results across these languages.

## introduction a variety of nlp tasks use taxonomic information, including question answering @xcite and information retrieval @xcite . taxonomies are also used as a resource for building knowledge and systematicity into neural models @xcite @xcite . nlp systems often retrieve taxonomic information from lexical databases such as word-net @xcite , which consists of taxonomies that contain semantic relations across many domains. while manually curated taxonomies provide useful information, they are incomplete and expensive to maintain @xcite . ## experiments in this section we describe the details of our datasets (section 3.1), and describe our evaluation metrics (section 3.2). we ran our experiments on a cluster with 10 quadro rtx 6000 gpus. each training runs finishes within one day on a single gpu. ## analysis in this section we analyze the models both quantitatively and qualitatively. unless stated otherwise, we analyze our model on the dev set and use roberta-large in the parenthood prediction step. ## related work taxonomy induction has been studied extensively, with both pattern-based and distributional approaches. typically, taxonomy induction involves hypernym detection, the task of extracting candidate terms from corpora, and hypernym organization, the task of organizing the terms into a hierarchy. while we focus on hypernym organization, many systems have studied the related task of hypernym detection. traditionally, systems have used patternbased features such as hearst patterns to infer hypernym relations from large corpora (e.g. @xcite @xcite . for example, @xcite propose a system that extracts pattern-based features from a corpus to predict hypernymy relations between terms. @xcite propose a system that similarly uses pattern-based features to predict hypernymy relations, in addition to harvesting relevant terms and using a graph-based longest-path approach to construct a legal taxonomic tree. later work suggests that, for hypernymy detection tasks, pattern-based approaches outperform those based on distributional models @xcite . subsequent work pointed out the sparsity that exists in pattern-based features derived from corpora, and showed that combining distributional and pattern-based approaches can improve hypernymy detection by addressing this problem @xcite . in this work we consider the task of organizing a set of terms into a medium-sized taxonomic tree. @xcite treat this as a structured learning problem and use belief propagation to incorpo-dataset model p r f1 science (averaged) ctp 29.4 28.8 29.1 @xcite 37.9 37.9 37.9 shang et al. (2020) rate siblinghood information. @xcite propose a reinforcement learning based approach that combines the stages of hypernym detection and hypernym organization. in addition to the task of constructing medium-sized wordnet subtrees, they show that their approach can leverage global structure to construct much larger taxonomies from the semeval-2016 task 13 benchmark dataset, which contain hundreds of terms @xcite . @xcite apply graph neural networks and show that they improve performance in constructing large taxonomies in the semeval-2016 task 13 dataset. another relevant line of work involves extracting structured declarative knowledge from pretrained language models. for instance, @xcite showed that a wide range of relations can be extracted from pretrained language models such as bert. our work differs in that we consider tree structures and incorporate web glosses. @xcite use pretrained models to generate explicit open-text descriptions of commonsense knowledge. other work has focused on extracting knowledge of relations between entities @xcite . blevins and zettlemoyer (2020) use a similar approach to ours for word sense disambiguation, and encode glosses with pretrained models. ## discussion our experiments show that pretrained language models can be used to construct taxonomic trees. importantly, the knowledge encoded in these pretrained language models can be used to construct taxonomies without additional web-based information. this approach produces subtrees with higher mean f 1 scores than previous approaches, which used information from web queries. when given web-retrieved glosses, pretrained language models can produce improved taxonomic trees. the gain from accessing web glosses shows that incorporating both implicit knowledge of input terms and explicit textual descriptions of knowledge is a promising way to extract relational knowledge from pretrained models. error analyses suggest specific avenues of future work, such as improving predictions for subtrees corresponding to abstractions, or explicitly modeling the global structure of the subtrees. experiments on aligned multilingual word-net datasets emphasize that more work is needed in investigating the differences between taxonomic relations in different languages, and in improving pretrained language models in non-english languages. our results provide strong baselines for future work on constructing taxonomies for different languages. ## ethical considerations while taxonomies (e.g., wordnet) are often used as ground-truth data, they have been shown to contain offensive and discriminatory content (e.g., @xcite . automatic systems created by pretrained language models can reflect and exacerbate the biases contained by their training corpora. more work is needed to detect and combat biases that arise when constructing and evaluating taxonomies. furthermore, we used previously constructed alignments to extend our results to wordnets in multiple languages. while considering english wordnet as the basis for the alignments allows for convenient comparisons between languages and is the standard method for aligning wordnets across languages, continued use of these alignments to evaluate taxonomy construction imparts undue bias towards conceptual relations found in english.