Distilling Hypernymy Relations from Language Models: On the Effectiveness of Zero-Shot Taxonomy Induction (2022)

Abstract: In this paper, we analyze zero-shot taxonomy learning methods which are based on distilling knowledge from language models via prompting and sentence scoring. We show that, despite their simplicity, these methods outperform some supervised strategies and are competitive with the current state-of-the-art under adequate conditions. We also show that statistical and linguistic properties of prompts dictate downstream performance.

## introduction taxonomy learning (tl) is the task of arranging domain terminologies into hierarchical structures where terms are nodes and edges denote is-a (hypernymic) relationships @xcite . domainspecific concept generalization is at the core of human cognition @xcite , and a key enabler in nlp tasks where inference and reasoning are important, e.g.: semantic similarity @xcite , wsd @xcite and, more recently, qa @xcite and nli @xcite . earlier approaches to taxonomy learning focused on mining lexico-syntactic patterns from candidate (hyponym, hypernym) pairs @xcite @xcite @xcite , clustering @xcite , graph-based methods @xcite or word embeddings @xcite . these methods, which largely rely on hand-crafted features, are still relevant today, and complement modern approaches exploiting language models (lms), either via sequence classification @xcite , or combining contextual, distributed, and lexico-syntactic features @xcite . in parallel, several works have recently focused on using lms as zero-shot tools for solving nlp tasks, e.g., commonsense, relational and analogical reasoning @xcite @xcite , multiword expression (mwe) identification @xcite , qa @xcite , domain labeling (sainz and rigau, 2021), or lexical substitution and simplification @xcite . moreover, by tuning and manipulating natural language queries (often referred to as prompts), impressive results have been recently obtained on tasks such as semantic textual similarity, entailment, or relation classification @xcite . in this paper, we evaluate lms on tl benchmarks using prompt-based and sentence-scoring techniques, and find not only that they are competitive with common approaches proposed in the literature (which are typically supervised and/or reliant on external resources), but that they achieve state-of-the-art results in certain domains. ## methodology we follow @xcite and define a prompt generation function τ p (t 1 , t 2 ) which maps a pair of terms and a prompt type p to a single sentence. for instance, τ kind ("physics", "science") = "physics is a kind of science" then, given a terminology t , the goal is to, given an input term @xmath0, retrieve its top k most likely hypernyms, (in our experiments, @xmath1, 3, 5}), using either masked language model (mlm) prompting ( §2.1), or sentence-scoring ( §2.2). ## experimental setup this section covers the datasets and prompts we use in our experiments foot_0 , as well as the different lms we consider. concerning evaluation metrics, we report standard precision (p ), recall (r) and f -score at the edge level @xcite . prompts we use the following prompts: language models we interrogate bert @xcite and roberta @xcite among masked lms, and gpt2 @xcite among causal lms. for each lm, we consider two variants corresponding to approximately 117m parameters and 345m parameters. ## results we also compare with ctp @xcite to illustrate the advantages of zero-shot methods vs finetuning. for the environment domain, we find that restrictmlm performs similar to ctp and lmscorer outperforms it. moreover, all 3 proposed approaches fail to outperform the other baselines. however, in science, all 3 of our approaches outperform ctp, while our best model (restrictmlm) outperforms taxi and is competitive with taxorl (ours has higher precison, but lower recall). note that compared to our zero-shot approaches, these methods are either supervised, expensive to train or take advantage of external taxonomical resources such as wordnet, or lexicosyntactic patterns mined from the web using different hand-crafted heuristics. we also show results for texeval-1's equipment and texeval-2's food ( ## analysis in this section, we provide an in-depth analysis of our approaches, including comparison of lms and statistical and semantic properties of prompts. lm comparison ## conclusion and future work we have presented a study of different lms under different settings for zero-shot taxonomy learning. compared with computationally expensive and highly heuristic methods, our zero-shot alternatives prove remarkably competitive. for the future, we could explore multilingual signals and the integration of traditional word embeddings with contextual representations.