ALL Dolphins Are Intelligent and SOME Are Friendly: Probing BERT for Nouns’ Semantic Properties and their Prototypicality (2021)

Abstract: Large scale language models encode rich commonsense knowledge acquired through exposure to massive data during pre-training, but their understanding of entities and their semantic properties is unclear. We probe BERT (Devlin et al., 2019) for the properties of English nouns as expressed by adjectives that do not restrict the reference scope of the noun they modify (as in “red car”), but instead emphasise some inherent aspect (“red strawberry”). We base our study on psycholinguistics datasets that capture the association strength between nouns and their semantic features. We probe BERT using cloze tasks and in a classification setting, and show that the model has marginal knowledge of these features and their prevalence as expressed in these datasets. We discuss factors that make evaluation challenging and impede drawing general conclusions about the models’ knowledge of noun properties. Finally, we show that when tested in a fine-tuning setting addressing entailment, BERT successfully leverages the information needed for reasoning about the meaning of adjective-noun constructions outperforming previous methods.

## introduction adjectival modification is one of the main types of composition in natural language @xcite . adjectives in attributive position 1 usually have a restrictive role on the reference scope of the noun they modify, limiting the set of things it refers to (e.g., white @xmath0. this property of adjectives has interesting entailment implications, generally leading to adjective-noun (an) constructions where the entailment relationship with the head noun holds (an |= n) @xcite . entailment is directional (white rabbit |= rabbit but rabbit |= white 1 adjectives that appear immediately before the noun they modify and form part of the noun phrase (e.g., white rabbit), as opposed to adjectives in predicative position that occur after the noun (e.g., this rabbit is white). masking properties singular a balloon is [mask]. plural balloons are [mask]. singular + usually a balloon is usually [mask]. plural + usually balloons are usually [mask]. singular+generally a balloon is generally [mask]. plural + generally balloons are generally [mask]. singular + can be a balloon can be [mask]. plural + can be balloons can be [mask]. most + plural most balloons are [mask]. all + plural all balloons are [mask]. some + plural some balloons are [mask]. masking quantifiers [mask] balloons are colourful. (all-most-some) [mask] balloons are large. (some-some-few) [mask] balloons are round. the notion of prototypicality has a prominent place in the computational linguistics literature, mainly by reference to relationships between nouns @xcite . the prototypicality of adjectives has been understudied and is absent from lexico-semantic resources such as wordnet @xcite and hyperlex @xcite . alongside the theoretical interest of this linguistic property and its impact on the entailment properties of an constructions, identifying prototypical adjectives has interesting practical implications. it can serve to retrieve information about the general concept (silk, blueberry) when queries include such an pairs (soft silk, small blueberry), or to discard adjectives that do not add new information about the noun they modify in summarisation or sentence compression. we investigate the knowledge that the bert model @xcite encodes about nouns' inherent properties as described in an constructions. although pre-trained language models have been shown to encode rich factual and commonsense knowledge @xcite , little is known about their understanding of the properties of the involved entities. we specifically explore whether bert encodes information about the prototypical properties of the class of objects denoted by a noun (for example, that all lobsters are red and blueberries are small). we use a set of collected norms that describe important concept features @xcite and their associated quantifiers @xcite . we rely on these data to derive cloze statements that we use to query bert about noun properties, and to train bert-based classifiers predicting these properties. we furthermore fine-tune bert for entailment and test it in a task that involves an constructions @xcite . our cloze task results show that bert has only marginal knowledge of noun properties and their prevalence, but can still successfully detect cases where the addition of an adjective does not alter the meaning of a sentence and where entailment is preserved. ## related work compositionality in an constructions has been a central topic in distributional and formal semantics. @xcite derive the meaning representation of a composite phrase from that of its constituents by performing algebraic operations (addition and multiplication) on distributional word semantic vectors, while @xcite and @xcite derive composite vectors through composition functions learned from corpus-harvested phrase vectors. in our work, we represent an phrases by combining the contextualised bert representations of a and n in sentences where they occur, using algebraic operations. we also investigate the extent to which the representations of a and n in an an phrase capture its meaning, since token-level bert embeddings encode information from the surrounding context. we furthermore address the entailment relationship between n and ans (n |= an). in the opposite direction (an |= n) entailment generally holds, i.e. almost all ans entail their head noun (red car |= car) @xcite . determining whether n |= an holds, however, depends on the semantic properties described by the adjective. we base this analysis on the addone dataset proposed by @xcite . addone consists of sentence pairs that contain an phrases annotated for entailment through crowdsourcing. this simplified entailment task differs from the classical rte task @xcite in that the premise and hypothesis differ by only one atomic edit (insertion of a). we use this task as a proxy for prototypicality based on the assumption that adjectives describing typical properties of a noun do not modify its scope (e.g., lobster |= red lobster, but car |= red car). prototypicality has been addressed in the literature mainly with respect to nouns, i.e. the typical hyponyms in a specific semantic class (e.g., dog |= animal), or member concepts that are most central to a category @xcite . @xcite also address verb prototypicality in terms of how typical of an action a verb is (e.g., "is to run a type of to move?"). our work extends this notion to adjectives describing noun properties in an phrases. on the probing side, previous work explores the factual and commonsense knowledge present in pretrained language models (lms). the lama (language model analysis) probe proposed by @xcite contains sets of facts from various knowledge sources. 2 each fact is converted into a "fill-in-the-blank" cloze statement that is used to query the lm for a missing token. a model is considered to know a fact ([subject, relation, object] triple) if it can successfully predict masked tokens in cloze statements expressing this fact (e.g., dante was born in ). the hasproperty relation in lama (extracted from conceptnet) is similar to our relation of interest as it links nouns to adjectives describing their properties. conceptnet contains 3,894 such pairs, but a close inspection of the data reveals several problematic cases (e.g., informal both, divine forgive, ten 10). additionally, the cloze statements proposed for this dataset were automati-cally extracted from open mind common sense (omcs) foot_1 sentences and are often too long, including irrelevant information that might confuse the model. 4 @xcite demonstrate the impact of prompt quality on lm probing but focus on relations involving encyclopedic knowledge (e.g., born/died in, profession, subclass). @xcite also explore the knowledge bert has about lexical, morphological and commonsense relations (e.g., hypernymy, meronymy, plural, cause-effect) through fine-tuning, but neither they address noun properties. herbelot and vecchi (2015) dataset (hvd) : hvd adds an extra level of quantification annotations to the mrd norms. three native speakers of english selected a natural language quantifier among [no, few, some, most, all] foot_3 for each concept-feature (c,f) pair, expressing the ratio of c instances having feature f (e.g., all guitars are musical instruments, but some guitars are electric). quantification is important for semantic inference; it can serve to understand set relations (such as synonymy and hyponymy) and to derive logically entailed sentences. we use the hvd dataset to probe bert for the prevalence of noun properties. pavlick and callison-burch (2016) dataset (addone) : the addone dataset is focused on an composition. it contains 5,560 sentence pairs involving an an pair (s n , s an ) which have been manually annotated for entailment (s n |= s an ) by crowd workers. addone sentence pairs differ by one atomic edit, the insertion of a (s n : "there are questions as to whether our culture has changed.", s an : "there are questions as to whether our traditional culture has changed."). sentences were collected from corpora of different genres and each pair was annotated with a score in a 5-point scale from 1 (contradiction) to 5 (entailment). only the pairs with high agreement (same score assigned by 2 out of 3 annotators) were retained. we use the addone dataset to assess bert's ability to detect entailment in an constructions. the dataset comes with a pre-defined split into training, development and test sets (83/10/7%) which we use in our experiments addressing entailment. ## cloze task experiments we use the bert-base-uncased and bert-large-uncased models pre-trained on the bookcorpus @xcite and on english wikipedia @xcite . the models are trained using a cloze task where tokens of the input sequence are masked and the models learn to fill the slots, and a binary classification objective where they need to predict whether a particular sentence follows a given sequence of words. ## classification experiments we probe bert representations for prototypicality in a classification setting, where models decide whether a describes a prototypical property of n. we use frozen embeddings (i.e. embeddings extracted from the pre-trained model) and fine-tuning. ## entailment in an constructions we fine-tune bert on addone to assess whether it captures the entailment relationship in an constructions. bert has shown high performance in other textual entailment tasks @xcite , but the addone dataset has proved challenging for other models relying on recurrent architectures. we follow @xcite and use addone for a binary classification task, with the labels entailment (for forward entailment and equivalence) and not entailment (encompassing the contradiction, independence and reverse entailment relations). similarly to the fine-tuning approach described in section 5.3, we feed into bert the two sentences in each pair (s n , s an ) separated by the special [sep] token. we again use the cls and tok classification heads. we fine-tune the model for 5 epochs with 0.1 dropout and select the learning rate based on the f1 score calculated over the actual entailment cases on the development set. 19 ## discussion retrieving prototypical knowledge about entities is a real challenge for distributional models, not necessarily because of the models themselves and how advanced they are, but because this information is rarely stated in texts. this is described in the literature as the "reporting bias" phenomenon @xcite which poses challenges on knowledge extraction. according to this phenomenon, rare actions or properties are overrepresented in texts at the expense of trivial ones. @xcite show that the generalisation capability of pre-trained language models allows them to better estimate the plausibility of frequent but unspoken actions, outcomes and properties than previous models, but that they also tend to overestimate that of the very rare, amplifying the bias that already exists in their training corpus. in this study, using methodology commonly used for probing contextual models, we have precisely explored whether retrieving knowledge about noun properties constitutes a challenge for these models, or whether they manage to retrieve this knowledge due to their impressive generalisation capabilities. since prototypical properties are often visual or perceptual, in future work we plan to combine text and visual features @xcite @xcite @xcite for retrieving noun properties, in order to see how bert can predict these properties when it has access to images alongside text. another goal is to collect evaluation data using cloze task queries in specifically designed crowdsourcing tasks. ## conclusion we have conducted a thorough investigation of the information encoded by bert about nouns' intrinsic properties. using datasets specifically compiled for psycholinguistics studies, we probed bert for noun properties and their prototypicality, as well as for the entailment relationship involving an constructions which indicates possible generalisations to the entire class denoted by the noun. our results show that information about noun properties, as described in word association norms, is hard to retrieve using cloze tasks. we discuss the limitations of semantic cloze tasks evaluation against existing resources, and the need for more flexible evaluation scenarios. however, knowledge about properties can still be leveraged by bert in a classification setting where the model is exposed to examples specifically encoding this information. we make our code and datasets available to promote further research in this direction. 20 yiben yang, larry birnbaum, ji-ping wang, and doug downey. 2018. extracting commonsense properties from embeddings with limited human guidance. in proceedings of the 56th annual meeting of the association for computational linguistics (volume 2: short papers), pages 644-649, melbourne, australia. association for computational linguistics.