Seeking Clozure: Robust Hypernym extraction from BERT with Anchored Prompts (2023)

Abstract: The automatic extraction of hypernym knowledge from large language models like BERT is an open problem, and it is unclear whether methods fail due to a lack of knowledge in the model or shortcomings of the extraction methods. In particular, methods fail on challenging cases which include rare or abstract concepts, and perform inconsistently under paraphrased prompts. In this study, we revisit the long line of work on pattern-based hypernym extraction, and use it as a diagnostic tool to thoroughly examine the hypernomy knowledge encoded in BERT and the limitations of hypernym extraction methods. We propose to construct prompts from established pattern structures: definitional (X is a Y); lexico-syntactic (Y such as X); and their anchored versions (Y such as X or Z). We devise an automatic method for anchor prediction, and compare different patterns in: (i) their effectiveness for hypernym retrieval from BERT across six English data sets; (ii) on challenge sets of rare and abstract concepts; and (iii) on consistency under paraphrasing. We show that anchoring is particularly useful for abstract concepts and in enhancing consistency across paraphrases, demonstrating how established methods in the field can inform prompt engineering.

## introduction semantic relations play a central role in knowledge representation @xcite and taxonomy construction @xcite . as the backbone of semantic relations, hyponymy/hypernymy relations express a hierarchical relation between a specific concept (the hyponym; e.g., dog) and a general one (the hypernym; e.g., mammal), and form the foundation of human concept understanding @xcite and relation reasoning @xcite lexico-syntactic prompt __ such as keas. anchored prompt __ such as keas and parrots. ## background we introduce the two approaches for hypernym extraction on which we build in this paper: patternbased ( § 2.1) and prompting plms ( § 2.2). ## anchored prompts we now introduce our framework of extracting hypernyms from a plm by constructing sets of prompts given a hyponym x and a pattern @xmath0, dfp +a , lsp, lsp +a }. we illustrate the workflow in figure 2 , with lsp +a as an example. prompt construction for each pattern type, we construct a set of prompts by instantiating each of its assigned patterns (cells in anchor extraction given x, we use bert to automatically extract a set of anchors, i.e., concepts z that share a hypernym with x. to acquire such anchors, we again adopt a set of established lexico-syntactic patterns that indicate the fact that x and z share a common hypernym @xcite @xcite . hypernym extraction being able to construct sets of prompts for vanilla (p dfp , p lsp ) and anchored prompts (p dfp +a , p lsp +a ), we are now in a position to prompt plms for hypernyms. separately for each prompt set p, 4 we score hypernym candidates y by their average probability across patterns p @xmath1: ## experimental setup datasets we conduct experiments on six english datasets. clsb @xcite and diag @xcite have been recently used 4 we drop subscripts to avoid clutter. to probe for hypernym knowledge in plms @xcite . the remaining four data sets are widely-used test sets for hypernym extraction more generally @xcite , namely bless @xcite , eval @xcite , leds @xcite , and shwartz @xcite . we only consider noun-noun hyponym-hypernym pairs from the datasets. dataset statistics are reported in ## anchor validation how accurate are the automatically mined anchors? we qualitatively and quantitatively inspect retrieved anchor concepts. we use wordnet for this purpose, and follow @xcite to consider a candidate z to be a valid anchor of x if they share a common ancestor, within two levels above x and four levels above z. we exclude hyponyms that are not in wordnet in this analysis. qualitative inspection reveals that retrieved anchors that are not wordnet siblings according to our definition above are often reasonable, see top 5 predicted anchors (z) car truck, motorcycle, boat, yes, bike apple grape, pear, nuts, vegetable, date train bus, plane, car, tram, truck corn bean, potato, barley, wheat, pea panzer tank, infantry, gun, artillery, panther motel hotel, yes, sure, restaurant, actually daisy rose, yes, lavender, rush, fern murre dog, bird, fox, crow, rabbit trireme warship, frigate, ship, ferry, battleship ## hypernym evaluation we first examine the effectiveness of lsp vs dfp and the added value of anchoring on our six data sets overall ( § 6.1). afterwards, we inspect specifically rare ( § 6.2) and abstract ( § 6.3) concepts as well as the well-known issue of inconsistency of responses in the face of prompt paraphrases ( § 6.4), explore different patterns in these contexts and end with an error analysis ( § 6.5). ## conclusion in this work, we bridge two powerful techniques in hypernym extraction: the pattern-based and prompt-based approach and use them as a diagnose tool to probe knowledge in bert. we provide a thorough study of how patterns from the corpus-mining literature can be used to probe neural models. we find that lsp and dfp exhibit similar capacities, while anchored patterns bring consistent and significant benefits, suggesting a way to overcome challenging scenarios. in particular, we demonstrated clear benefits for rare hyponyms and abstract hypernyms, and an increase in the reliability of retrieved hypernyms under paraphrased prompts. this finding can direct future work on prompt design to extract robust and consistent hypernyms knowledge. the idea of anchoring prompts can be extended to other semantic relations such as part-of and synonyms to advance taxonomy induction and knowledge graph construction.