What do Deck Chairs and Sun Hats Have in Common? Uncovering Shared Properties in Large Concept Vocabularies (2023)

Abstract: Concepts play a central role in many applications. This includes settings where concepts have to be modelled in the absence of sentence context. Previous work has therefore focused on distilling decontextualised concept embeddings from language models. But concepts can be modelled from different perspectives, whereas concept embeddings typically mostly capture taxonomic structure. To address this issue, we propose a strategy for identifying what different concepts, from a potentially large concept vocabulary, have in common with others. We then represent concepts in terms of the properties they share with the other concepts. To demonstrate the practical usefulness of this way of modelling concepts, we consider the task of ultra-fine entity typing, which is a challenging multi-label classification problem. We show that by augmenting the label set with shared properties, we can improve the performance of the state-of-the-art models for this task.

## introduction various applications rely on knowledge about the meaning of concepts, which is typically encoded in the form of embeddings. for instance, pretrained concept embeddings are used to provide prior knowledge about the labels in few-shot and zero-shot learning tasks @xcite @xcite @xcite @xcite . concept embeddings also play a central role in the area of knowledge management, for instance for taxonomy learning @xcite , knowledge graph alignment @xcite , ontology alignment @xcite and ontology completion @xcite . in such applications, language models (lms) such as bert @xcite cannot be used directly, since we need concept embeddings which do not depend on sentence context. for this reason, several authors have proposed strategies for obtaining decontextualised (or static) concept embeddings from lms @xcite @xcite @xcite @xcite . decontextualised concept embeddings tend to primarily reflect basic taxonomic structure, e.g. capturing the fact that televisions are electronic devices and that salmon are fish. however, applications often rely on different facets of meaning. in computer vision applications, we want concept embeddings to reflect visual features; e.g. they should capture the fact that chess boards, televisions and books have something in common (i.e. being rectangular). when modelling scene graphs @xcite , we may rather want embeddings that capture which objects are often found together in the same visual scene. for instance, concept embeddings should then reflect the fact that televisions, sofas and rugs have something in common (i.e. that they are typically found in the living room). when modelling food ontologies, concept embeddings should perhaps capture the fact that salmon and walnuts share the property of being rich in omega-3. in principle, if sufficient training data is available, we could learn application-specific concept embeddings by fine-tuning a language model. however, the role of concept embeddings is often precisely to address the scarcity of application-specific training data. we, therefore focus on strategies that use pre-trained models in an unsupervised way. concept embeddings can be viewed as compact representations of similarity metrics. however, what matters in the aforementioned applications is often less about modelling similarity than about modelling what different concepts have in common. this is an important distinction because we can list the properties that a concept satisfies in an application-independent way, which is not possi- property concepts used for staying connected telephone number, google, area code, user name, facebook, land line, email, call center, web server, link farm, name server used for beach activities deck chair, beach volleyball, beach ball, sun hat, sun dog ... a round shape bubble, small ball, ball bearing, sphere, disc, round top, centre circle, wheel, tin can, cone, cylinder, oval, tube located in cities parking lot, high street, suburbs, supermarket, mall, food court, piazza, city hall, cinema, general store used for communication address book, twitter, voice message, instant message, voice mail letter bomb, listening post, mail bomb, ring tone, street address, wireless operator accommodation options for camping beer tent, man cave, rock shelter, holiday camp, camp, mobile home, guest room, inn, resort, trading post, cabin, cabin boy, hunting lodge, lodge, mess hall accounting expense account, transaction, register, business intelligence, finance, business record, income statement, backup, census, consumer credit, estimate, evaluation, listing, maintenance, market research tv series genres detective, soap opera, reality show, crime, mystery, cartoon blockbuster, hero, radio drama, season finale, series finale aches fever, strain, head cold, battle fatigue, sore, pain, hurt, fatigue bow shock, cold, dry eye, hunger, obesity, sting, stretch mark adulthood middle age, thirty, age group, womanhood, drinking age, aged, careers, elder, generation, maturity, autumn, bachelor, birth, birthplace, childhood, decade, era advertisements 'slogan', 'brand image', 'publicity', 'advertising', 'product placement', 'banner ad', 'advertiser', 'advertisement', 'advance', 'propaganda', 'ambush marketing', 'dot product', 'fan mail', 'fan service' baby carriers basket, baby seat, basket case, car seat, carrier, baby book, baby bottle, body bag to implement this strategy, we need an efficient mechanism for identifying these commonalities. the vocabulary is often too large to directly use large language models (llms) to solve this task. instead, we rely on a bi-encoder architecture to efficiently retrieve the properties that are satisfied by each concept @xcite . once the initial set of properties has been retrieved, we use a fine-tuned deberta model @xcite to filter these properties. after summarising the related work in section 2, we describe our strategy for uncovering shared properties in section 3. our subsequent evaluation in section 4 focuses on two tasks. first, we carry out an intrinsic evaluation to show the effectiveness of the proposed filtering strategy. second, we demonstrate the practical usefulness of uncovering shared properties on the downstream task of ultra-fine entity typing. ## related work the task of predicting commonsense properties has been studied by several authors @xcite @xcite @xcite . modelling the commonalities between concepts, or conversely, identifying outliers, has also been previously considered. for instance, outlier detection has been used for intrinsic evaluation of word embeddings @xcite @xcite . however such benchmarks are focused on taxonomic categories. moreover, we focus on finding commonalities in large vocabularies, containing perhaps tens of thousands of concepts. another related task is entity set expansion @xcite @xcite . given an initial set of entities, e.g. {france, germany, italy}, this task requires selecting other entities that have the same properties (e.g. being european countries). however, the focus is usually on named entities, whereas we focus on concepts. ## identifying shared properties let v be the considered vocabulary of concepts. our aim is to associate each concept @xmath0. to find suitable properties, we first retrieve a set of candidate properties for each concept in v using an efficient bi-encoder model. subsequently, we verify the retrieved properties with a joint encoder. retrieving candidate properties for each concept @xmath1, we want to find a set of properties that are likely to be satisfied by that concept. the vocabulary v may contain tens of thousands of concepts, which makes it expensive to use an llm for this purpose. we instead rely on the bi-encoder model from @xcite . the idea is to fine-tune two bert models: one for learning concept embeddings (ϕ con ) and one for learning property embeddings (ϕ prop ). the probability that concept c has property p is then estimated as σ ϕ con (c) • ϕ prop (p) , with σ the sigmoid function. to train this model, we need examples of concepts and the properties they satisfy. the only largescale knowledge base that contains such training examples is conceptnet, which is unfortunately rather noisy and imbalanced. @xcite therefore trained their model on a large set of (hyponym,hypernym) pairs from microsoft concept graph @xcite , together with examples from genericskb @xcite . since the performance of the bi-encoder heavily depends on the training data, and existing training sets are sub-optimal, we created a training set of 109k (concept,property) pairs using chatgpt foot_1 . simply asking chatgpt to enumerate the properties of some concept tends to result in verbose explanations describing overly specific properties, which are less helpful for identifying commonalities. therefore, we used a prompt which specifically asked chatgpt to identify properties that are shared by several concepts. we also experimented with a training set we derived from conceptnet 5.5. specifically, we converted instances of the relations isa, partof, locatedat, usedfor and hasproperty into a set of 63,872 (concept,property) pairs. the full details can be found in the appendix. ## conclusions concept embeddings are often used to provide prior knowledge in applications such as (multi-label) few-shot learning. we have proposed an alternative to the use of embeddings for such applications, where each concept is instead represented in terms of the properties it satisfies. our motivation comes from the observation that concept embeddings tend to primarily capture taxonomic relatedness, which is not always sufficient. we first use a bi-encoder to efficiently retrieve candidate properties for each concept, and then use a joint encoder to decide which properties are satisfied. the resulting property assignments have allowed us to improve the state-of-the-art in ultra-fine entity typing.