Subspace Representations for Soft Set Operations and Sentence Similarities (2024)

Abstract: In the field of natural language processing (NLP), continuous vector representations are crucial for capturing the semantic meanings of individual words. Yet, when it comes to the representations of sets of words, the conventional vector-based approaches often struggle with expressiveness and lack the essential set operations such as union, intersection, and complement. Inspired by quantum logic, we realize the representation of word sets and corresponding set operations within pre-trained word embedding spaces. By grounding our approach in the linear subspaces, we enable efficient computation of various set operations and facilitate the soft computation of membership functions within continuous spaces. Moreover, we allow for the computation of the F-score directly within word vectors, thereby establishing a direct link to the assessment of sentence similarity. In experiments with widely-used pre-trained embeddings and benchmarks, we show that our subspace-based set operations consistently outperform vector-based ones in both sentence similarity and set retrieval tasks.

## introduction embedding-based word representations have become fundamental in the field of natural language processing (nlp). models like word2vec @xcite and glove @xcite , along with recent transformer-based architectures @xcite , have underscored the significance of embeddings in capturing the complexities of linguistic semantics. the importance of representing collections of words is pivotal in understanding concepts and * work done while at nara institute of science and technology. in response to these challenges, our study introduces a novel methodology that exploits the principles of quantum logic @xcite , applied within embedding spaces to define set operations. our proposed framework adopts a subspace-based approach for representing word sets, aiming to maintain the intricate semantic relationships within these sets. we represent a word set as a subspace which is spanned by pretrained embeddings. additionally, it adheres to the foundational laws of set theory as delineated in the framework of quantum logic. this compliance ensures that our set operations, such as union, intersection, and complement, are not only mathematically robust but also linguistically meaningful when applied them in pre-trained embedding space. we first introduce a subspace set representation along with basic operations (∩, ∪, @xmath0. subsequently, to highlight the usefulness of our proposed framework, we introduce two core set computations: text similarity and set membership. the empirical results consistently point towards the notable superiority of our approach; our straightforward approach of spanning subspaces with pretrained embedding sets enables a rich set representation, and we demonstrated its consistent performance enhancement in downstream tasks (figure 1 ). our research contributions include: ## symbolic set operations we first formulate various set operations in a pretrained embedding space. among many types of operations for practical nlp applications, this work focuses on set similarity: @xmath1, boy, walks, in, this, park }, @xmath2, kid , runs, in, the, square}, @xmath3, blue, green, orange, . . . }, for this purpose, we need following representations on a pre-trained embedding space foot_0 : an element and a set of elements the representations of an element and a set of elements are the most basic ones. to exploit word embeddings, we represent a word (e.g., orange) as an element and a group of words (e.g., {red , blue, green, orange, . . . }) as a word set. quantification of set membership (indicator function) membership denotes a relation in which word w is an element of set a, i.e., @xmath4. we quantify it based on vector representations. although the membership is typically a binary decision identical to that in a symbolic space, it can also be measured by the degree of closeness in a continuous vector space. membership can be computed as an indicator function. the indicator function similarity between discrete symbol sets set similarity, such as recall and precision, is an essential operation when calculating the similarity of texts. despite its simplicity, the word overlapbased sentence similarity serves as a remarkably effective approximation and has found widespread practical application, as evidenced by numerous studies @xcite @xcite . they stand out as excellent similarity metrics based on embeddings. bertscore @xcite , which utilizes embeddings for its computation, is grounded in recall and precision foot_1 . the typical computations for recall (r) and precision (p ) are as follows foot_2 : basic set operations we need three basic set operations: intersection (a ∩ b ), union (a ∪ b ), and complement ( a ). they allow us to represent various sets using such different combinations as color ∩ fruit. ## subspace-based set representations we propose the representations of a word set and set operations based on quantum logic @xcite . they hold advantages of geometric properties in an embedding space, and the set operations are guaranteed to hold for the laws of a set defined in quantum logic. ## text concept set retrieval task in this section, we evaluate the capability of our proposed set operations (∩, ∪, @xmath5. task we evaluate our set operations by the set expansion task introduced by @xcite . in this task, the model is given a set of words that share a common concept or theme. the objective is to expand this set by retrieving relevant words from a vocabulary that fit the same concept. for instance, if the initial set includes words like "apple", "banana", and "peach", the task would be to identify and add other fruit names (e.g., "orange") to this set. for the evaluation, we follow @xcite . we report recall (r@k) and median, that indicate whether the words in the test set can be ranked higher. embeddings we used the most standard pretrained word embeddings in all of our experiments: 300-dimensional glove foot_4 @xcite , which was pre-trained with common crawl, and 300-dimensional word2vec foot_5 @xcite , which was pre-trained with google news. ## related work symbol-based similarities between word sets have been proposed, such as jaccard coefficient @xcite @xcite and tf-idf-based similarity @xcite . unfortunately, symbol-based methods cannot capture the semantic similarity of similar sets or words when the symbols are different. while many studies have explored representing word sets in pre-trained embedding spaces @xcite , they primarily focus on set similarity. our approach, however, extends beyond this by developing a comprehensive framework for various set operations within these spaces. utilizing subspace properties, our method not only represents word sets but also performs a range of versatile operations, such as calculating textual similarities and membership degrees. many methods for learning the representation of sets have been proposed because of the wide range of possible applications @xcite @xcite @xcite . in contrast, our approach does not require additional training. this enables us to compute set representations and operations using popular general-purpose language models, which are trained on the general domain @xcite . ## conclusion this study introduces a novel framework for set representation and operations within pre-trained embedding spaces, employing linear subspaces grounded in quantum logic. this approach extends the scope of conventional embedding set operations by incorporating vector-based representations.