FUSE : Measure-Theoretic Compact Fuzzy Set Representation for Taxonomy Expansion (2024)

Abstract: Taxonomy Expansion, which relies on modeling concepts and concept relations, can be formulated as a set representation learning task. The generalization of set, fuzzy set, incorporates uncertainty and measures the information within a semantic concept, making it suitable for concept modeling. Existing works usually model sets as vectors or geometric objects such as boxes, which are not closed under set operations. In this work, we propose a sound and efficient formulation of set representation learning based on its volume approximation as a fuzzy set. The resulting embedding framework, Fuzzy Set Embedding , satisfies all set operations and compactly approximates the underlying fuzzy set, hence preserving information while being efficient to learn, relying on minimum neural architecture. We empirically demonstrate the power of FUSE on the task of taxonomy expansion, where FUSE achieves remarkable improvements up to 23% compared with existing baselines. Our work marks the first attempt to understand and efficiently compute the embeddings of fuzzy sets.

## introduction taxonomy is a crucial data structure for modeling semantic concepts, hence of great importance for nlp @xcite @xcite . concepts in a taxonomy can often be viewed as sets, the most fundamental object in mathematics, whose operations directly link to first order logic (fol). for example, in a science taxonomy, "biology" and "computer science" are semantic concepts, whose intersection results in a new concept "bio-informatics", and "diffusion model" and "gan" belong to a coarser-grained concept, "generative model". usually, sets are seen as a fixed collection of objects. for example, the set n consists numbers {0, 1, • • • } by definition. however, in the context of semantic concepts, their meanings can change overtime and incorporate ambiguity. for example, "beauty" is a concept that has become broader overtime, and "deep learning models" can expand to have more elements with more discoveries made by the community. this underlying uncertainty and ambiguity are instead captured by a fuzzy set @xcite , an extension of classical sets. a wide range of works have been developed for set representation learning. early efforts are made to construct simple vector embeddings @xcite @xcite @xcite based on similarity measures. to better model complex relationships such as asymmetrical relationships between concepts, geometric embeddings @xcite @xcite have been developed, which leverages the inherent geometric properties to model hierarchical relationships. however, these methods cannot address all the set operations including intersection, union, and complement. for example, box embedding @xcite @xcite doesn't define union and complement of boxes. worse yet, existing geometric objects are not closed under set operations: the union of two boxes is no necessarily a box, which can compromise the consistency of reasoning in the embedding space. in this paper, we directly tackle the challenge of fuzzy set representation learning for concept modeling. our objective is to use their volume to quantify their information and their associated uncertainty. however, learning powerful representations for fuzzy sets is challenging. first, although extensive efforts have been made to incorporate deep learning techniques into fuzzy set modeling @xcite @xcite , their training procedure could be expensive when the universe of discourse is large. second, compared with geometric embeddings, which have clear definitions of volume, it is unclear how to model the volumes of fuzzy sets due to the introduction of uncertainty and their abstract nature. to tackle the previous challenges, we propose a principled and learnable model named fuzzy set embedding (fuse) for fuzzy set representation learning. the core of fuse is to introduce a compact approximation of fuzzy sets and then prove that fuse can arbitrarily approximate the original fuzzy sets under reasonable regularity conditions. fuse avoids the computational burden of accounting for all the elements in the space of discourse at once, while enjoying the properties of fuzzy logic, hence satisfying all set operations. we further introduce a rank-based loss and asymmetric relations to enhance set representation learning. to validate the effectiveness of our proposed fuse, we evaluate on taxonomy expansion task and show that fuse can achieve the performance improvement up to 23% compared with state-of-the-art baselines, and we explore the effectiveness of our theoretical formulation through various ablation studies. our main contributions can be summarized as follows @xcite we propose an embedding framework to model fuzzy sets and show that the embeddings satisfy all set operations and are closed under set operations. (b) we systematically construct this embedding as a proper approximation of fuzzy sets. (c) we demonstrate the effectiveness of this embedding framework on taxonomy expansion by comparing it against previous vector and geometrybased embedding methods. ## proposed framework: fuse we now present fuzzy set embedding (fuse) for learning set representations in a principled way. ## conclusion for taxonomy expansion, we propose a novel and theoretically sound fuzzy set embedding (fuse) to model concepts and relationship between concepts that incorporate set operations @xcite . we show theoretically that fuse preserves the information of the fuzzy set with sufficiently fine-grained partitions and demonstrate empirically that it can outperform existing vector-based and geometry-based embedding methods on taxonomy expansion. for future works, we believe that expanding the taxonomy dataset with more complicated combination of set operations, such as first order logic (fol), can further improve the model performance.