Modelling Commonsense Properties Using Pre-Trained Bi-Encoders (2022)

Abstract: Grasping the commonsense properties of everyday concepts is an important prerequisite to language understanding. While contextualised language models are reportedly capable of predicting such commonsense properties with human-level accuracy, we argue that such results have been inflated because of the high similarity between training and test concepts. This means that models which capture concept similarity can perform well, even if they do not capture any knowledge of the commonsense properties themselves. In settings where there is no overlap between the properties that are considered during training and testing, we find that the empirical performance of standard language models drops dramatically. To address this, we study the possibility of fine-tuning language models to explicitly model concepts and their properties. In particular, we train separate concept and property encoders on two types of readily available data: extracted hyponym-hypernym pairs and generic sentences. Our experimental results show that the resulting encoders allow us to predict commonsense properties with much higher accuracy than is possible by directly fine-tuning language models. We also present experimental results for the related task of unsupervised hypernym discovery.

## introduction pre-trained language models @xcite have been found to capture a surprisingly rich amount of knowledge about the world @xcite . focusing on commonsense knowledge, @xcite used bert to predict whether a given concept (e.g. teddy bear) satisfies a given commonsense property (e.g. is dangerous). to this end, they convert the input into a simple sentence (e.g. "a teddy bear is dangerous") and treat the task as a standard sentence classification task. remarkably, they found this approach to surpass human performance. @xcite moreover found that language models can, to some extent, capture commonsense properties that are rarely expressed in text, thus mitigating the issue of reporting bias that has traditionally plagued initiatives for learning commonsense knowledge from text @xcite . despite these encouraging signs, however, modelling commonsense properties remains highly challenging. a key concern is that language models are typically fine-tuned on a training set that contains the same properties as those in the test set. for instance, the test data from @xcite includes the question whether peach has the property eaten in summer, while the training data asserts that apple, banana, orange and pear all have this property. to do well on this task, the model does not actually need to capture the knowledge that peaches are eaten in summer; it is sufficient to capture that peach is similar to apple, banana, orange and pear. for this reason, we propose new training-test splits, which ensure that the properties occurring in the test data do not occur in the training data. our experiments show that the ability of language models to predict commonsense properties drops dramatically in this setting. our aim is to develop a strategy for modelling the commonsense properties of concepts. given the limitations that arise when language models are used directly, a natural approach is to pre-train a language model on some kind of auxiliary data. unfortunately, resources encoding the commonsense properties of concepts tend to be prohibitively noisy. to illustrate this point, we also experiment with genericskb (bhakthavatsalam et al., 2020), a large collection of generic sentences (e.g. "coffee contains minerals and antioxidants which help prevent diabetes"), to obtain concept-property pairs for pre-training. given such pre-training data, we then train a concept encoder φ con and a property encoder φ prop such that σ(φ con (c) • φ prop (p)) indicates the probability that concept c has property p. in summary, our main contributions are as follows: (i) we propose a new evaluation setting which is more realistic than the standard benchmarks for predicting commonsense properties; (ii) we analyse the potential of hypernymy datasets and generic sentences to act as pre-training data; and (iii) we develop a simple but effective bi-encoder architecture for modelling commonsense properties. ## related work several authors have analysed the extent to which language models such as bert capture commonsense knowledge. as already mentioned, @xcite evaluated the ability of bert to predict commonsense properties from the mcrae dataset @xcite , which we also use in our experiments. the same dataset was used by @xcite to analyse whether bert-based language models could generate concept names from their associated properties; e.g. given the input "a ⟨mask⟩ has fur, is big, and has claws", the model is expected to predict that ⟨mask⟩ corresponds to the word bear. conversely, apidianaki and garí soler (2021) considered the problem of generating adjectival properties from prompts such as "mittens are generally ⟨mask⟩". note that the latter two works evaluated pre-trained models directly, without fine-tuning, whereas the experiments @xcite involved fine-tuning the language model on a task-specific training set first. when the main motivation is to probe the abilities of language models, avoiding fine-tuning has the advantage that any observed abilities reflect what is captured by the pre-trained language model itself, rather than learned during the fine-tuning phase. however, @xcite argue that the extent to which pre-trained language models capture commonsense knowledge is limited, suggesting that some form of fine-tuning is essential in practice. interestingly, this remains the case for large language models. for instance, their model had 7 billion parameters, while @xcite report that the predictions from gpt-3 @xcite had to be filtered by a so-called critic model when distilling a commonsense knowledge graph. the strategy taken by comet @xcite is to fine-tune a gpt model @xcite on triples from commonsense knowledge graphs. being based on an autoregressive language model, comet can be used to predict concepts that take the form of short phrases, which is often needed when reasoning about events (e.g. to express motivations or effects). however, as illustrated in a final line of related work concerns the modelling of hypernymy. several authors have proposed specialised embedding models for this task @xcite . most relevant to our work, @xcite fine-tune a bert-based language model to predict the validity of a concept-hypernym pair. inspired by the effectiveness of hearst patterns @xcite , they use prompts of the form "[hypernym] such as [concept]" (and similar for other hearst patterns). the extent to which the pre-trained bert model captures hypnernymy has also been studied. for instance, hanna and @xcite use prompts where the prediction of the ⟨mask⟩ token can be interpreted as the prediction of a hypernym, to avoid the need for fine-tuning the model. ## methodology let a set of concept-property pairs k be given, where (c, p) ∈ k means that concept c is asserted to have the property p. we write c and p for the sets of concepts and properties in k, i.e. @xmath0, p) ∈ k} and @xmath1, p) ∈ k}. we use the term "property" in a broad sense, covering both semantic attributes, as in the pair (banana, sweet), and hypernyms, as in the pair (banana, fruit). this is motivated by the fact that hypernyms often encode knowledge about semantic attributes, as in the pair (banana, sweet fruit). in particular, our hypothesis is that, by treating hypernyms and semantic attributes in a unified way, we can pre-train a model on readily available hypernym datasets and use it to predict semantic attributes. we want to train a model that can predict for a given pair (c, p) whether c has property p. two general strategies can be pursued when developing such models. the first strategy is to use a socalled cross-encoder, which amounts to fine-tuning a single language model to predict whether a given input (c, p) represents a valid pair or not. the second strategy is to use a so-called bi-encoder, which amounts to the idea that c and p are separately encoded, with the resulting vectors then being used to predict whether (c, p) is a valid pair. in this paper, we pursue the latter strategy. this is primarily motivated by the fact that the concept and property encoders enable a wider range of applications. a cross-encoder can only be used to predict whether a given pair (c, p) is valid or not. in contrast, a biencoder model can also be used to efficiently find the properties p of a given concept c. moreover, the resulting concept and property embeddings may themselves be useful as static representations of word meaning, e.g. as label embeddings for zeroshot or few-shot learning @xcite @xcite @xcite . finally, bi-encoders can be trained more efficiently than cross-encoders. datasets to train our model, we need a large set of concept-property pairs k. unfortunately, high-quality knowledge of this kind is not readily available. part of the underlying issue is that properties of concepts are rarely explicitly stated in text, which is why directly using information extraction techniques is not straightforward. however, initiatives for extracting hypernyms from text have been much more successful, starting with the seminal work of @xcite . a key observation is that fine-grained hypernyms often express commonsense properties, typically as a mechanism for refining hypernyms that would otherwise be too broad. for instance, microsoft concept graph @xcite lists vitamin c rich food as a hypernym of strawberry, as a refinement of the more general hypernym food. by pre-training our model on concept-hypernym pairs, we may thus expect it to learn about commonsense properties as well. to directly test this hypothesis, we use a set of such concept-hypernym pairs as our pre-training set k. specifically, we collect the 100k concepthypernym pairs from microsoft concept graph foot_2 with the highest confidence score 4 we will refer to this dataset as mscg. as a second strategy, we attempt to convert the mscg dataset into a set of concept-property pairs. to this end, we look for pairs (c, h 1 ) and (c, h 2 ) where h 2 is a suffix of h 1 . specifically, if h 1 = mh 2 and m is an adjectival phrase, then we assume that m describes a property of c. for instance, mscg contains the pairs (strawberry, vitamin c rich food) and (strawberry, food). based on this, we would include the pair (strawberry, vitamin c rich) in k. clearly this is a heuristic strategy, which may produce non-sensical or misleading pairs. for instance, according to mscg, strawberry is a lowsugar berry, but this does not entail that strawberry has the property low-sugar in general. however, we may expect most of the pairs that are generated with this strategy to be meaningful. a total of 8186 concept-property pairs were obtained in this way. we refer to the resulting dataset as prefix. finally, going beyond concept-hypernym pairs, we derive a dataset from genericskb @xcite , which contains generic sentences such as "bananas are an important food staple in the tropics". due to the regular structure of such sentences, we can easily convert them into concept-property pairs; e.g. the aforementioned sentence would become (banana, important food staple in the tropics). we collect a set of 100k such pairs, by processing the sentences with the highest confidence (i.e. the ones which are most likely to be generic sentences) whose length is at most 7. the reason why we focus on shorter sentences is because they are more likely to capture salient information. we refer to this dataset as gkb. training objective given the pairs in k, we pretrain two encoders, φ con and φ prop , using binary cross-entropy. in particular, the loss function for a given mini-batch is defined as follows: here k batch represents the subset of k that is in the current mini-batch. for efficiency reasons, we sample these mini-batches as follows. first, we sample a subset c batch of k concepts from c. then, for each concept c in c batch we sample one property @xmath2, p) ∈ k. let p batch be the set of properties that are thus obtained. the sets of positive examples k batch and negative examples n batch are then defined as follows: in other words, the positive examples are the pairs from k that involve a concept from c batch and a property from p batch . the negative examples are all the other pairs that we can form by taking a concept from c batch and a property from p batch . this in-batch negative sampling strategy ensures that after encoding |c batch | concepts and |p batch | properties, we can take |c batch |×|p batch | training examples into account. given that the encoders φ con and φ prop correspond to fine-tuned language models, and the encoding steps are thus time-consuming, in-batch negative sampling enables a significant speed-up compared to naive strategies in which positive and negative examples are sampled independently. similar strategies are commonly used in neural information retrieval @xcite . ## experiments in our experiments, we primarily focus on commonsense property classification, i.e. predicting whether some concept has a given property. we also demonstrate the usefulness of the concept and property encoders on the task of hypernym discovery. finally, we also present a qualitative analysis. training details we pre-train the concept and property encoders on the datasets introduced in section 3. we also consider variants in which these datasets are combined; e.g. we write mscg+prefix for the dataset combining the pairs from mscg and prefix. to pre-train our model, we construct separate validation data in the same way. in particular, for mscg, we select the validation split by taking the next 10k most confident pairs from microsoft concept graph (i.e. after removing the pairs from the mscg dataset itself), and similar for the other datasets. we train the model for 100 epochs, using early stopping with a patience of 20. we use the adamw optimizer @xcite with a learning rate of 2e-6 and set the batch size to 8. we use bert-base-uncased as our default language model @xcite , although we have also experimented with bert-large-uncased, roberta-base and roberta-large @xcite . ## conclusions we studied the problem of modelling the commonsense properties of concepts. we argued that the standard evaluation setting does not faithfully assess the extent to which models capture knowledge about commonsense properties, and proposed two new evaluation settings. these new settings were found to be highly challenging for language models, with performance being close to random. we furthermore found that pre-training a bi-encoder model on hypernymy data or generic sentences can lead to substantial performance gains. however, there remains a lot of room for further improvements, which will likely require novel insights.