Modelling Commonsense Commonalities with Multi-Facet Concept Embeddings (2024)

Abstract: Concept embeddings offer a practical and efficient mechanism for injecting commonsense knowledge into downstream tasks. Their core purpose is often not to predict the commonsense properties of concepts themselves, but rather to identify commonalities, i.e. sets of concepts which share some property of interest. Such commonalities are the basis for inductive generalisation, hence high-quality concept embeddings can make learning easier and more robust. Unfortunately, standard embeddings primarily reflect basic taxonomic categories, making them unsuitable for finding commonalities that refer to more specific aspects (e.g. the colour of objects or the materials they are made of). In this paper, we address this limitation by explicitly modelling the different facets of interest when learning concept embeddings. We show that this leads to embeddings which capture a more diverse range of commonsense properties, and consistently improves results in downstream tasks such as ultra-fine entity typing and ontology completion.

## introduction many knowledge engineering tasks require knowledge about the meaning of concepts. as a motivating example, let us consider the problem of ontology expansion, which consists in uncovering properties of, and relationships between concepts, given the names of these concepts and an initial knowledge base. despite the popularity of large language models (llms), the use of concept embeddings remains attractive in such settings @xcite @xcite . indeed, using llms directly is often impractical and highly inefficient, as ontologies can involve tens of thousands of concepts. concept embeddings can also be integrated with structural features more easily @xcite , for instance by using them to initialise the node embeddings of a graph neural network (gnn). concept embeddings similarly play an important role in many multi-label classification tasks, especially in the zero-shot or few-shot regime @xcite @xcite @xcite . as a representative example of such a task, we will consider the problem of ultra-fine entity typing @xcite , which consists in assigning semantic types to mentions of entities, where a total number of around 10k candidate types are considered. in such tasks, the role of pre-trained concept embeddings is to inject prior knowledge about the meaning of the type labels @xcite . note that we cannot straightforwardly accomplish this with llms, as they have been found to struggle with information extraction tasks @xcite . moreover, scalability is often an important concern for information extraction systems, which further complicates the use of llms. we take the view that concept embeddings, in the aforementioned applications, are primarily needed to capture commonalities among the concepts involved. for ontology expansion, this is true by definition, since the task explicitly involves identifying sets of concepts that have some property in common. for ultra-fine entity typing, @xcite reported that directly using pre-trained label embeddings was challenging. instead, they proposed to cluster the set of labels based on pre-trained concept embeddings, and to use the resulting clusters to structure the label space. the idea of using embeddings to structure the label space also lies at the heart of many traditional approaches for zero-shot and few-shot classification. a key limitation of traditional concept embeddings comes from the fact that they primarily reflect basic taxonomic categories. for instance, the embedding of banana is typically similar to that of other fruits, but dissimilar from the embeddings of other yellow things. some authors have pro-posed to learn multi-facet embeddings as a way of alleviating these concerns @xcite @xcite . essentially, rather than learning a single vector representation of each concept (or entity), they learn a fixed number of different vectors, each focusing on a different facet. however, learning such representations is challenging for two main reasons. first, learning multi-facet representations requires some kind of supervision signal about the facets of interest @xcite , which is not readily available for many domains. second, existing approaches consider a fixed set of facets, which makes them unsuitable for open-domain settings. indeed, the facets of interest strongly depend on the nature of the concepts involved. when modelling food, we may be interested in embeddings that capture their nutritional content. when modelling household appliances, we may want a representation that captures where in the house they are typically found. rather than using the same set of facets for all concepts, we thus need a more dynamic representation framework. in this paper, we propose a novel method for learning multi-facet concept embeddings based on two key ideas. first, we rely on chatgpt foot_0 to collect a diverse set of (property, facet) pairs, such as (yellow, colour), (found in the kitchen, location) or (sweet, taste), allowing us to treat the problem of learning multi-facet embeddings as a supervised learning problem. second, rather than learning several independent vector representations, we only learn a single embedding for each concept, treating facets instead as masks on the set of coordinates. this approach offers several modelling advantages, including the fact that facets can have a hierarchical structure (e.g. colour is a sub-facet of appearance) and the fact that we do not have to tune the number and dimension of the facets a priori. specifically, we train three bert @xcite encoders: one encoder to map concepts onto their embedding, one encoder to map properties onto their embedding, and one encoder to map properties onto the embedding of the corresponding facet. we show that these encoders can be effectively trained using only training data obtained from chatgpt, although the best results are obtained by augmenting this training data with examples from conceptnet foot_1 . ## related work concept embeddings the idea that language models of the bert family can be used for learning concept embeddings has been studied extensively. some approaches simply use the name of the concept as input to the bert encoded, possibly together with a short prompt @xcite @xcite . other approaches instead use contextualised representations from sentences mentioning the concept, selected from some corpus @xcite @xcite @xcite . these approaches have been developed with different motivations in mind. one common motivation is to learn something about the language model itself by inspecting the resulting concept embeddings, such as biases @xcite or the model's grasp of lexical semantics @xcite . other authors have rather focused on the use of embeddings for predicting semantic properties of concepts @xcite @xcite . our paper can be seen as a continuation of this latter research line, where we aim to improve the range of properties that can be captured by concept embeddings through the use of facet embeddings. commonalities @xcite recently argued that the main purpose of concept embeddings, when it comes to downstream applications, is usually to identify what different concepts have in common. specifically, given a set of concepts, they first used the corresponding concept embeddings to predict a set of properties for each concept. the resulting predictions were then filtered using a natural language inference (nli) model. finally, properties that were found for at least two concepts where identified as shared properties. they showed, on the task of ultra-fine entity typing, that by augmenting the training data with these shared properties, models were able to generalise better. this idea also relates to the notion of conceptualisation @xcite . essentially, the latter works have suggested to augment commonsense knowledge graphs by generalising the concepts involved. this often involves replacing a specific concept (e.g. a football game) by a description referring to some salient property (e.g. a relaxing event). @xcite showed that the resulting generalisations of commonsense knowledge graphs were useful for zero-shot com-monsense question answering. the aforementioned methods all rely on the availability of a set of properties (or hypernyms) that can be used to generalise a given set of concepts. in practice, however, it is hard to obtain comprehensive property sets, which means that many commonalities may not be discovered. moreover, certain commonalities are hard to describe, even though they intuitively make sense. 3 to avoid such limitations, we identify commonalities by clustering concept embeddings. ## proposed approach we propose a bi-encoder based concept embedding model which is capable of representing concepts w.r.t. different facets. the key stumbling block in previous work on learning multi-facet embeddings has been the difficulty in acquiring a meaningful supervision signal about which properties belong to the same facet (e.g. that large and small both refer to size). as explained in section 3.1, we can now overcome this difficulty by collecting propertyfacet pairs from llms. our proposed model itself is described in section 3.2. finally, we explain how facet-specific embeddings can be extracted once the model has been trained (section 3.3). ## experiments we analyse the effectiveness of the proposed multifacet concept embedding model. we intrinsically evaluate the embeddings on predicting commonsense properties (section 4.1) and outlier detection (section 4.2). we furthermore consider two downstream applications: ontology completion (section 4.3) and ultra-fine entity typing (section 4.4). foot_5 ## conclusions many applications rely on background knowledge about the meaning of concepts. what is needed often boils down to knowledge about the commonalities between different concepts, as this forms the basis for inductive generalisation. clustering pre-trained concept embeddings has been proposed in previous work as a viable strategy for modelling such commonalities. however, the resulting clusters primarily capture taxonomic categories, while commonalities that depend on various commonsense properties are essentially ignored. in this paper, we proposed a simple strategy for obtaining more diverse representations, by taking into account different facets of meaning when training the concept encoder. we found that the resulting concept representations lead to consistently better results, across all the considered tasks.