Distilling Hypernymy Relations from Language Models: On the Effectiveness of Zero-Shot Taxonomy Induction
Abstract
In this paper, we analyze zero-shot taxonomy learning methods which are based on distilling knowledge from language models via prompting and sentence scoring.
We show that, despite their simplicity, these methods outperform some supervised strategies and are competitive with the current state-of-the-art under adequate conditions.
We also show that statistical and linguistic properties of prompts dictate downstream performance.
Introduction
Taxonomy learning (TL) is the task of arranging domain terminologies into hierarchical structures where terms are nodes and edges denote is-a (hypernymic) relationships (Hwang et al., 2012).
Domainspecific concept generalization is at the core of human cognition (Yu et al., 2015), and a key enabler in NLP tasks where inference and reasoning are important, e.g.: semantic similarity (Pilehvar et al., 2013; Yu and Dredze, 2014), WSD (Agirre et al., 2014) and, more recently, QA (Joshi et al., 2020) and NLI (Chen et al., 2020).
Earlier approaches to taxonomy learning focused on mining lexico-syntactic patterns from candidate ( hyponym, hypernym ) pairs (Hearst, 1992; Snow et al., 2004; Kozareva and Hovy, 2010; Boella and
Di Caro, 2013; Espinosa-Anke et al., 2016), clustering (Yang and Callan, 2009), graph-based methods (Fountain and Lapata, 2012; Velardi et al., 2013) or word embeddings (Fu et al., 2014; Yu et al., 2015).
These methods, which largely rely on hand-crafted features, are still relevant today, and complement modern approaches exploiting language models (LMs), either via sequence classification (Chen et al., 2021), or combining contextual, distributed, and lexico-syntactic features (Yu et al., 2020).
In parallel, several works have recently focused on using LMs as zero-shot tools for solving NLP tasks, e.g., commonsense, relational and analogical reasoning (Petroni et al., 2019; Bouraoui et al., 2020;
Ushio et al., 2021; Paranjape et al., 2021), multiword expression (MWE) identification (EspinosaAnke et al., 2021; Garcia et al., 2021), QA (Shwartz et al., 2020; Banerjee and Baral, 2020), domain labeling (Sainz and Rigau, 2021), or lexical substitution and simplification (Zhou et al., 2019).
Moreover, by tuning and manipulating natural language queries (often referred to as prompts ), impressive results have been recently obtained on tasks such as semantic textual similarity, entailment, or relation classification (Shin et al., 2020;
Qin and Eisner, 2021).
In this paper, we evaluate LMs on TL benchmarks using prompt-based and sentence-scoring techniques, and find not only that they are competitive with common approaches proposed in the literature (which are typically supervised and/or reliant on external resources), but that they achieve SoTa results in certain domains.
Methodology
We follow Ushio et al.
and define a prompt generation function τ p ( t 1 , t 2 ) which maps a pair of terms and a prompt type p to a single sentence.
For instance, τ kind (“physics”, “science”) =
“physics is a kind of science”
Then, given a terminology T , the goal is to, given an input term t ∈ T , retrieve its top k most likely hypernyms, (in our experiments, k ∈ { 1 , 3 , 5 } ), using either masked language model (MLM) prompting ( § 2.1), or sentence-scoring ( § 2.2).
2. MLM Prompting RestrictMLM
Petroni et al.
(2019) introduced a “fill-in-the-blanks” approach based on cloze statements (or prompts ) to extract relational knowledge from pretrained LMs.
The intuition being that an LM can be considered to “know” a fact (in the form of a < subject, relation, object > triple) such as < Madrid, capital-of, Spain > if it can successfully predict the correct words when queried with prompts such as “Madrid is the capital of [MASK] ”.
We extend this formulation to define a hypernym retrieval function f R ( · ) as follows:
[Equation]
where p is a prompt type, and T is a one-hot encoding of the terms T in the LM’s vocabulary.
We follow previous works (Petroni et al., 2019; Kassner et al., 2021) and restrict the output probability distribution since this task requires the construction of a lexical taxonomy starting from a fixed vocabulary.
PromptMLM
For completeness, we also report results for an unrestricted variant of RestrictMLM , where the LM’s entire vocabulary is considered.
2. LMScorer
Factual (and true) information such as “Trout is a type of fish” should be scored higher by a LM than fictitious information such as “Trout is a type of mammal”.
The method for scoring a sentence depends on the type of LM used.
Causal Language Models
Given a sentence W , causal LMs ( C ) predict token w i using only past tokens W <i .
Thus, a likelihood score can be estimated for each token w i from the LM’s next token prediction.
The corresponding scores are then aggregated to yield a score for the sentence W .
[Equation]
Masked Language Models
Given a sentence W , masked LMs ( M ) replace w i by [MASK] and predict it using past and future tokens.
Thus, a pseudolikelihood score can be computed for each token w i by iteratively masking it and using the LM’s masked token prediction (Wang and Cho, 2019; Salazar et al., 2020).
The corresponding scores are then aggregated to yield a score for the sentence W .
[Equation]
Given the above, we can cast TL as a sentencescoring problem by evaluating the natural fluency of hypernymy-eliciting sentences.
Specifically, for each term t , we score the sentences generated using τ p ( · ) with every other term t ′ in the terminology.
We then select the term-pair with the highest sentence score and assume that the corresponding term t ′ is a hypernym of t .
Formally, we define a hypernym selection function f S ( · ) as follows:
[Equation]
where s refers to the scoring function determined by the LM used.
Experimental setup
This section covers the datasets and prompts we use in our experiments 1 , as well as the different
LMs we consider.
Concerning evaluation metrics, we report standard precision ( P ), recall ( R ) and
F -score at the edge level (Bordea et al., 2016).
Dataset Details
We evaluate our proposed approaches on datasets belonging to two TL SemEval tasks ( TExEval-1 , Bordea et al.
and
TExEval-2 , Bordea et al.
Following recent literature, we consider the equipment taxonomy from TExEval-1 and the English-language environment , science and food taxonomies from TExEval2 .
For the science taxonomy, our results are based on an average of the 3 subsets , which is in line with previous work.
Since these datasets do not come with training data, they are well suited for unsupervised approaches.
[Table]
Eurovoc
[Table]
Combined
[Table]
Prompts
We use the following prompts:
• gen.
[ t 2 ] is more general than [ t 1 ].
• spec.
: [ t 1 ] is more specific than [ t 2 ].
type : [ t 1 ] is a type of [ t 2 ]
gen. and spec.
prompts are hand-crafted templates to encode, in a general way, the hypernymy relationship.
The choice of the type prompt, however, comes from a set of experiments involving all
LPAQA (Jiang et al., 2020) prompts under the “ is a subclass of ” category.
We do not consider automatic prompt generation techniques (Shin et al., 2020) due to the absence of training data.
Note that for each prompt, we replace t 1 with the input term so that the task is always to predict its hypernym.
Language Models
We interrogate BERT (Devlin et al., 2019) and RoBERTa (Liu et al., 2019) among masked LMs, and GPT2 (Radford et al., 2019) among causal LMs.
For each LM, we consider two variants corresponding to approximately 117M parameters and 345M parameters.
Results
Table 2 shows the results on TExEval-2 ’s science and environment .
We compare with the current state of the art ( Graph2Taxo ) (Shang et al., 2020), as well as with other strong baselines such as TaxoRL (Mao et al., 2018) and TAXI (Panchenko et al., 2016), the highest ranked system in TExEval-2 .
We also compare with CTP (Chen et al., 2021) to illustrate the advantages of zero-shot methods vs finetuning.
For the environment domain, we find that RestrictMLM performs similar to CTP and LMScorer outperforms it.
Moreover, allproposed approaches fail to outperform the other baselines.
However, in science , all 3 of our approaches outperform CTP , while our best model ( RestrictMLM ) outperforms TAXI and is competitive with TaxoRL (ours has higher precison, but lower recall).
Note that compared to our zero-shot approaches, these methods are either supervised, expensive to train or take advantage of external taxonomical resources such as WordNet, or lexicosyntactic patterns mined from the web using different hand-crafted heuristics.
We also show results for TExEval-1 ’s equipment and TExEval-2 ’s food (Table 3).
Both datasets are considerably larger than environment and science .
We compare with the corresponding highest ranked system, namely TAXI for food , and INRIASAC (Grefenstette, 2015) for equipment .
For both domains, all 3 of our approaches outperform the corresponding TExEval best-performing systems.
This suggests that zero-shot TL with LMs is robust,
[Table]
easily scalable and feasible on large taxonomies.
Finally, a clear bottleneck for prompt-based methods is that only single-token terms can be predicted (using a single [MASK] token), making this approach a lower bound for TL.
[Table]
[Table]
Analysis
In this section, we provide an in-depth analysis of our approaches, including comparison of LMs and statistical and semantic properties of prompts.
LM Comparison
Table 4 compares the best configuration for each LM.
We can immediately see that a conservative approach (i.e., k = 1 with the type prompt) almost always yields the best
F -score.
Another important conclusion is that, among MLMs, BERT-Large performs best across the board, with BERT generally outperforming
RoBERTa, a finding in line with previous works (Shin et al., 2020).
Concerning causal LMs, GPT2 Medium outperforms its smaller counterpart as well as both MLMs for sentence-scoring.
Sensitivity to Prompts
There is interested in understanding the model’s sensitivity to prompts and whether frequency can explain downstream performance in lexical semantics tasks (Chiang et al., 2020).
In the context of prompt vs. performance
[Table]
correlation, we find that prompt-based downstream performance on TL can be attributed to: (1) syntactic completeness and (2) semantic correctness.
For (1), we find that prompts that are syntactically more complete (e.g., “ [X] is a type of [Y] ” vs “ [X] is a type [Y] ”, the difference being the prepositional phrase) perform better.
For (2), we find that prompts that unambiguously encode hypernymy are also better (i.e., the type prompt, as opposed to other noise-inducing templates such as “ is a ” or “ is kind of ”).
Finally, out of the cleanest prompts, the most frequent in pretraining corpora are the most competitive.
Table 5 confirms the intuition that the type prompt is not only unambiguous, but also highly frequent when compared to similar (noisefree and syntactically complete) prompts.
[Table]
Single-Token vs Multi-Token Hypernyms
Table 6 compares F-score on original terminology vs filtered terminology, where filtered terminology contains only the terms that have single-token hypernyms.
The results show that % Increase in Fscore is inversely proportional to the % Retained.
This can be explained by the fact that smaller % of terms retained implies higher % of multi-token hypernyms in the original dataset that cannot be predicted using prompting.
Thus, the increase in Fscore by removing such hypernyms should increase as the % Retained decreases.
[Table]
Conclusion and Future Work
We have presented a study of different LMs under different settings for zero-shot taxonomy learning.
Compared with computationally expensive and highly heuristic methods, our zero-shot alternatives prove remarkably competitive.
For the future, we could explore multilingual signals and the integration of traditional word embeddings with contextual representations.