W ord2 B ox: Capturing Set-Theoretic Semantics of Words using Box Embeddings (2022)

Abstract: Learning representations of words in a continuous space is perhaps the most fundamental task in NLP, however words interact in ways much richer than vector dot product similarity can provide. Many relationships between words can be expressed set-theoretically, for example, adjective-noun compounds (eg. “red cars”⊆“cars”) and homographs (eg. “tongue”∩“body” should be similar to “mouth”, while “tongue”∩“language” should be similar to “dialect”) have natural set-theoretic interpretations. Box embeddings are a novel region-based representation which provide the capability to perform these set-theoretic operations. In this work, we provide a fuzzy-set interpretation of box embeddings, and learn box representations of words using a set-theoretic training objective. We demonstrate improved performance on various word similarity tasks, particularly on less common words, and perform a quantitative and qualitative analysis exploring the additional unique expressivity provided by Word2Box.

## introduction the concept of learning a distributed representation for a word has fundamentally changed the field of natural language processing. the introduction of efficient methods for training vector representations of words in word2vec @xcite , and later glove @xcite as well as fast-text @xcite revolutionized the field, paving the way for the recent wave of deep architectures for language modeling, all of which implicitly rely on this fundamental notion that a word can be effectively represented by a vector. while now ubiquitous, the concept of representing a word as a single point in space is not particularly natural. all senses and contexts, levels of abstraction, variants and modifications which the word may represent are forced to be captured by interdum et males u ad a fames ac an t e ipsum primis in faucibus. et iam rhoncus eu tellus non pellentesque. pellentesque sollicitudin nulla el i t , at gravida es t laoreet a. sed dignissim in libero a laoreet. integer vitae purus id leo fringilla mat t is id at lacus. nunc ac quam ut risus sollicitudin lacinia vitae a velit. nulla facilisi. phasellus sit amet tempor ex. aliquam erat volutpat. ut consectetur arc u vitae turpis gravida, a placerat dui laoreet. curabitur maximu s vehicula leo ac commodo. aliquam erat volutpat. nulla nec lectus in diam mat t is vehicula. nulla facilisi. maecenas tempus scelerisque aliq u am. vivamus lobortis lobortis viverra. proin vulputate neque at tempor vehicula. aliquam placerat urna sit amet porttitor mat t is . morbi eget mat t is justo. lorem ipsum dolor sit amet , consectetur ad ip is c in g el i t . aenean et mat t is nunc. donec feugiat nibh eu ultrices mo les t ie. nullam at arc u cursus, commodo nibh eget , suscipit ipsum. duis congue nec quam ac aliq u am. sus pendis s e arc u odio, ullamcorper eget sapien ut, sagittis aliq u et felis. class ap t en t taciti sociosqu ad litora torquent per conubia nostra, per inceptos himenaeos. cras eges t as non lacus ut ullamcorper. proin tincidunt velit nec es t au c t o r, vitae ac c u ms an nisi fermentum. fusce ac sem tempor, pellentesque es t ut, lacinia justo. aenean at au gu e quis purus fringilla finibus condimentum at purus. aenean id rutrum magn a, ac volutpat turpis. in finibus sed nunc vitae tempus. proin consequat lacinia tellus pharetra consequat. donec nisi turpis, pharetra id risus eges t as , mo les t ie pellentesque el i t . quisque non justo dictum, consectetur dui ornare, feugiat nisi. morbi ac consectetur en i m. nullam luctus dignissim en i m, ac tincidunt mas s a hendrerit vitae. ut ultrices eu purus eu fringilla. aenean tristique mau ris ultricies posuere porttitor. nunc placerat a dolor nec eu i s mo d . aliquam vestibulum pretium es t non lacinia. phasellus ut el ei f en d el i t . nullam venenatis au gu e quam, id lacinia leo porta a. cras odio turpis, ac c u ms an quis faucibus eget , imperdiet et urna. nulla fermentum pretium au c t o r. nulla at sem ut nisl faucibus scelerisque. the specification of a single location in euclidean space. it is thus unsurprising that a number of alternatives have been proposed. gaussian embeddings @xcite propose modeling words using densities in latent space as a way to explicitly capture uncertainty. poincaré embeddings @xcite attempt to capture a latent hierarchical graph between words by embedding words as vectors in hyperbolic space. trained over large corpora via similar unsupervised objectives as vector baselines, these models demonstrate an improvement on word similarity tasks, giving evidence to the notion that vectors are not capturing all relevant structure from their unsupervised training objective. a more recent line of work explores regionbased embeddings, which use geometric objects such as disks @xcite , cones @xcite @xcite , and boxes @xcite to represent entities. these models are often motivated by the need to express asymmetry, benefit from particular inductive biases, or benefit from calibrated probabilistic semantics. in the context of word representation, their ability to represent words using geometric objects with well-defined intersection, union, and difference operations is of interest, as we may expect these operations to translate to the words being represented in a meaningful way. in this work, we introduce word2box, a region-based embedding for words where each word is represented by an n-dimensional hyperrectangle or "box". of the region-based embeddings, boxes were chosen as the operations of intersection, union, and difference are easily calculable. specifically, we use a variant of box embeddings known as gumbel boxes, introduced in @xcite . our objective (both for training and inference) is inherently set-theoretic, not probabilistic, and as such we first provide a fuzzy-set interpretation of gumbel boxes yielding rigorously defined mathematical operations for intersection, union, and difference of gumbel boxes. we train boxes on a large corpus in an unsupervised manner with a continuous bag of words (cbow) training objective, using the intersection of boxes representing the context words as the representation for the context. the resulting model demonstrates improved performance compared to vector baselines on a large number of word similarity benchmarks. we also compare the models' abilities to handle set-theoretic queries, and find that the box model outperforms the vector model 90% of the time. inspecting the model outputs qualitatively also demonstrates that word2box can provide sensible answers to a wide range of set-theoretic queries. ## fuzzy sets of windows in this section, we describe the motivation for using fuzzy sets to represent words, starting with an approach using traditional sets. first, given a word @xmath0, we can consider the windows centered at v, cen w (v) := {@xmath1: cen(w) = v}, and the set of windows whose context contains v, con w (v) := {@xmath2: con(w) ∋ v}. note that cen w is a function which takes in a word and returns a set of windows, whereas cen is a function which takes in a window and returns the center word, and a similar distinction holds for con w and con. a given window is thus contained inside the intersection of the sets described above, namely as an example, the window @xmath3, is contained inside the cen w ("fox") set, as well as con w ("quick"), con w ("brown"), con w ("jumps"), con w ("over"). with this formulation, the intersection of the con w sets provide a natural choice of representation for the context. we might hope that cen w (v) provides a reasonable representation for the word v itself, however by our set theoretic definition for any u ̸ = v we have we would like the representation of u to overlap with v if u has "similar meaning" to v, i.e. we would like to consider cen w (v) := {@xmath4: cen(w) similar to v}. a crisp definition of meaning or similarity is not possible @xcite due to individual subjectivity. inter-annotator agreement for @xcite is only 0.67, for example, which makes it clear that cen w (v) could not possibly be represented as a traditional set. instead, it seems natural to consider cen w (v) as represented by a fuzzy set (w, m), where m(w) ∈ [0, 1] can be thought of as capturing graded similarity between v and cen(w). 2 in the same way, we can define con w (v) := {@xmath5: con(v) ∋ w similar to v}, which would also be represented as a fuzzy set. 3 as we wish to capture these similarities with a machine learning model, we now must find trainable representations of fuzzy sets. remark 1. our objective of learning trainable representations for these sets provides an additional practical motivation for using fuzzy sets -namely, the hard assignment of elements to a set is not differentiable. any gradient-descent based learning algorithm which seeks to represent sets will have to consider a smoothed variant of the characteristic function, which thus leads to fuzzy sets. ## gumbel boxes as fuzzy sets in this section we will describe how we model fuzzy sets using gumbel boxes @xcite . as noted in section 2.2, the gumbel box model represents entities @xmath6 since this is contained in [0, 1], we have that (r d , p (@xmath7. for clarity, we will refer to this fuzzy set as box f (x). the set complement operation has a very natural interpretation in this setting, as box f (x) c has membership function 1 -p (@xmath8, that is, the probability of z not being inside the gumbel box. the product t-norm is a very natural choice as well, as the intersection box f (x) ∩ box f (y) will have membership function p (@xmath9, which is precisely the membership function associated with box g (x) ∩ box g (y), where here the intersection is between gumbel boxes as defined in @xcite . finally, we find that the membership function for the union box f (x) ∪ box f (y) is given (via the t-conorm) by remark 2. prior work on gumbel boxes had not defined a union operation on gumbel boxes, however (3) has several pleasing properties apart from being a natural consequence of using the product t-norm. first, it can be directly interpreted as the probability of z being inside box g (x) or box g (y). second, if the gumbel boxes were embedded in a space of finite measure, as in @xcite , integrating (3) would yield the probability corresponding to p (box(x) ∪ box(y)). to calculate the size of the fuzzy set box f (x) we integrate the membership function over r d , the connection between this integral and that which was approximated in dasgupta et al. ( 2020 ) is provided by lemma 3 of @xcite , and thus we have by exploiting linearity and closure under intersection, it is possible to calculate the size of arbitrary fuzzy intersections, unions, and set differences, as well as any combination of such operations. remark 3. if our boxes were embedded in a space of finite measure, as in @xcite , the sizes of these fuzzy sets would correspond to the intersection, union, and negation of the binary random variables they represent. ## training in this section we describe our method of training fuzzy box representations of words, which we refer to as word2box. in section 3 we defined the fuzzy sets cen w (v) and cen w (v), and in section 4 we established that gumbel boxes can be interpreted as fuzzy sets, thus for word2box we propose to learn center and context box representations given a window, @xmath10, . . . , w 0 , . . . , w j ], we noted that w must exist in the intersection, and thus we consider a max-margin training objective where the score for a given window is given as to create a negative example w ′ we follow the same procedure as cbow from @xcite , replacing center words with a word sampled from the unigram distribution raised to the 3/4. we also subsample the context words as in @xcite . as a vector baseline, we compare with a word2vec model trained in cbow-style. we attach the source code with supplementary material. ## experiments and results we evaluate both word2vec and word2box on several quantitative and qualitative tasks that cover the aspects of semantic similarity, relatedness, lexical ambiguity, and uncertainty. following the previous relevant works @xcite @xcite , we train on the lemmatized wackypedia corpora @xcite , specifically ukwac which is an english language corpus created by web crawling. after additional pre-processing (details in appendix a) the corpus contains around 0.9 billion tokens, with just more than 112k unique tokens in the vocabulary. noting that an n-dimensional box actually has 2n parameters (for min and max coordinates), we compare 128-dimensional word2vec embeddings and 64dimensional word2box embeddings for all our experiments. we train over 60 different models for both the methods for 10 epochs using random sampling on a wide range of hyperparameters (please refer to appendix c for details including learning rate, batch size, negative sampling, sub-sampling threshold, etc.). in order to ensure that the only difference between the models was the representation itself, we implemented a version of word2vec in pytorch, including the negative sampling and subsampling procedures recommended in @xcite , using the original implementation as a reference. as we intended to train on gpu, however, our implementation differs from the original in that we use stochastic gradient descent with varying batch sizes. we provide our source code at https://github.com/iesl/word2box . ## related work learning distributional vector representations from a raw corpus was introduced in @xcite , quickly followed by various improvements @xcite . more recently, vector representations which incorporate contextual information have shown significant improvements @xcite @xcite . as these models require context, however, word2vec-style approaches are still relevant in settings where such context is unavailable. hyperbolic representations @xcite @xcite have become popular in recent years. most related to our setting, @xcite propose a hyperbolic analog to glove, with the motivation that the hyperbolic embeddings will discover a la- ## conclusion in this work we have demonstrated that box embeddings can not only effectively train to represent pairwise similarity but also can capture the rich set-theoretic structure of words via unsupervised training. this is a consequence of the fact that gumbel boxes are an efficient parameterization of fuzzy sets, with sufficient representational capacity to model complicated co-occurrance interactions while, at the same time, allowing for tractable computation and gradient-based training of set-theoretic queries. the set-theoretic representation capabilities of box models allow them to generalize in a calibrated manner, leading to a more coherent and self-consistent model of sets.