Learning Lexical Subspaces in a Distributional Vector Space (2020)

Abstract: In this paper, we propose LexSub, a novel approach towards unifying lexical and distributional semantics. We inject knowledge about lexical-semantic relations into distributional word embeddings by defining subspaces of the distributional vector space in which a lexical relation should hold. Our framework can handle symmetric attract and repel relations (e.g., synonymy and antonymy, respectively), as well as asymmetric relations (e.g., hypernymy and meronomy). In a suite of intrinsic benchmarks, we show that our model outperforms previous approaches on relatedness tasks and on hypernymy classification and detection, while being competitive on word similarity tasks. It also outperforms previous systems on extrinsic classification tasks that benefit from exploiting lexical relational cues. We perform a series of analyses to understand the behaviors of our model.1Code available at https://github.com/aishikchakraborty/LexSub .

## introduction pre-trained word embeddings are the bedrock of modern natural language processing architectures. this success of pre-trained word embeddings is attributed to their ability to embody the distributional hypothesis @xcite , which states that ''the words that are used in the same contexts tend to purport similar meanings'' @xcite . the biggest strength of the embedding methodstheir ability to cluster distributionally related words-is also their biggest weakness. this contextual clustering of words brings together words that might be used in a similar context in the text, but that might not necessarily be semantically similar, or worse, might even be antonyms @xcite . several techniques have been proposed in the literature to modify word vectors to incorporate lexical-semantic relations into the embedding space @xcite @xcite @xcite @xcite . the common theme of these approaches is that they modify the original distributional vector space using auxiliary lexical constraints to endow the vector space with a sense of lexical relations. however, a potential limitation of this approach is that the alteration of the original distributional space may cause a loss of the distributional information that made these vectors so useful in the first place, leading to degraded performance when used in the downstream tasks. this problem could be further exacerbated when multiple relations are incorporated, especially as different lexical-semantic relations have different mathematical properties. for example, synonymy is a symmetric relation, whereas hypernymy and meronymy are asymmetric relations. it would be difficult to control the interacting effects that constraints induced by multiple relations could have on the distributional space. the solution that we propose is to enforce a separation of concerns, in which distributional information is addressed by a central main vector space, whereas each lexical relation is handled by a separate subspace of the main distributional space. the interface between these components is then a projection operation from the main distributional space into a lexical subspace. our framework, lexsub, thus formulates the problem of enforcing lexical constraints as a problem of learning a linear subspace for each of the lexical relations within the distributional vector space. figure 1 shows a conceptual diagram of the relationship between the distributional space and the lexical subspaces in lexsub. we show that lexsub outperforms previous methods in a variety of evaluations, particularly on intrinsic relatedness correlation tasks, and in extrinsic evaluations in downstream settings. we also show that lexsub is competitive with existing models on intrinsic similarity evaluation tasks. we run a series of analyses to understand why our method improves performance in these settings. our experimental results suggest that explicitly separating lexical relations into their own subspaces allows the model to better capture the structure of each lexical relation without being polluted by information from the distributional space. conversely, the main distributional vector space is not polluted by the need to model lexical relations in the same space, as is the case for previous models. furthermore, the explicit linear projection that is learned ensures that a relation-specific subspace exists in the original distributional vector space, and can thus be discovered by a downstream model if the extrinsic task requires knowledge about lexical-semantic relations. contributions. in summary, we propose lexsub, a framework for learning lexical linear subspaces within the distributional vector space. the proposed framework can model all major kinds of lexical-semantic relations, namely, attract-sym-metric, repel-symmetric, and attract-asymmetric. we demonstrate that our approach outperforms or is competitive with previous approaches on intrinsic evaluations, and outperforms them on a suite of downstream extrinsic tasks that might benefit from exploiting lexical relational information. finally, we design a series of experiments to better understand the behaviors of our model and provide evidence that the separation of concerns achieved by lexsub is responsible for its improved performance. post-hoc approaches. in the post-hoc approach, pre-trained word vectors such as glove @xcite , word2vec @xcite , fasttext @xcite , or paragram @xcite are fine-tuned to endow them with lexical relational information @xcite @xcite @xcite @xcite @xcite . in this paper, we primarily discuss lexsub as a post-hoc model. this formulation of lexsub is similar to the other post-hoc approaches mentioned above with the significant difference that the lexical relations are enforced in a lexical subspace instead of the original distributional vector space. @xcite explores the idea of learning specialized subspaces with to reduce the dimensionality of distributional space such that it maximally preserves relevant task-specific information at the expense of distributional information. unlike @xcite , our proposed method tries to retain the distributional information in the embeddings so that they can be used as a general-purpose initialization in any nlp pipeline. embeddings from @xcite 's method can only be used for the task on which they were trained. ad-hoc approaches. the ad-hoc class of approaches add auxiliary lexical constraints to the distributional similarity loss function, usually, a language modeling objective like cbow @xcite or recurrent neural network language model @xcite . these constraints can either be viewed as a prior or as a regularizer to the distributional objective @xcite @xcite @xcite . in other work, the original language modeling objective is modified to incorporate lexical constraints @xcite @xcite @xcite @xcite . we discuss the ad-hoc formulation of lexsub in appendix a. an alternate axis along which to classify these approaches is by their ability to model different types of lexical relations. these types can be enumerated as symmetric-attract (synonymy), symmetric-repel (antonymy), and asymmetricattract (hypernymy, meronymy). most approaches mentioned above can handle symmetric-attract type relations, but only a few of them can model other types of lexical relations. for example, @xcite al. (2015) our proposed framework can model all types of lexical relations, namely, symmetric-attract, symmetric-repel, and asymmetric-attract, and uses of all four major lexical relations found in lexical resources like wordnet, namely, synonymy, antonymy, hypernymy, and meronymy, and could flexibly include more relations. to our knowledge, we are the first to use meronymy lexical relations. other approaches. several approaches do not fall into either of the categories mentioned above. a subset of these approaches attempts to learn lexical relations, especially hypernymy, directly by embedding a lexical database, for example, poincaré embeddings @xcite or order-embeddings @xcite . another set of approaches, like dih @xcite or word2gauss @xcite attempt to learn the hypernymy relation directly from the corpus without relying on any lexical database. the third set of approaches attempt to learn a scoring function over a sparse bag of words (sbow) features. these approaches are summarized by @xcite . ## training setup in this section, we describe the datasets and models that we use in our experiments. the output of our model is the main vector space embedding that is endowed with the specialized lexical subspaces. all our evaluations are done on the main vector space embeddings unless stated otherwise. ## baselines vanilla. the vanilla baselines refer to the original glove word embeddings without any lexical constraints. retrofitting. retrofitting @xcite uses similarity constraints from lexical resources to pull similar words together. the objective function that retrofitting optimizes consists of a reconstruction loss l dist and a symmetric-attract loss l syn att-sym with @xmath0, w proj @xmath1, and counterfitting. counterfitting @xcite builds up on retrofitting but also support repel symmetric relations. their objective function consists of three parts: synonym attract, antonym repel, and a vector space preservation loss, similar to l syn att-sym , l syn rep-sym , and l dist , respectively. lear. lear @xcite expands the counterfitting framework by adding a lexical entailment (le) loss. this le loss encodes a hierarchical ordering between concepts (hyponym-hypernym relationships) and can handle attract asymmetric relations. we train each of the baseline models using the lexical resources described in section 4.2. lear, lexsub, and counterfitting were trained on all four lexical relations whereas the retrofitting was trained only on attract relations, namely, synonymy, hypernymy, and meronymy. this is due to retofitting's inability to handle repel type relations. we also report the results of our experiments with lexsub and the baselines trained on the lexical resource from lear in appendix b. hypernymy tasks. following roller et al. (2018), we consider three tasks involving hypernymy: graded hypernymy evaluation, hypernymy classification, and directionality detection. we use the hypernymy subspace embeddings for lexsub for these experiments. for graded hypernymy evaluation, we use the hyperlex dataset @xcite and report the results on the complete hyperlex dataset. we measure spearman's ρ between the cosine similarity of embeddings of the word pairs and the human evaluations. the hypernymy classification task is an unsupervised task to classify whether a pair of words are hypernym/hyponym of each other. we consider four of the five benchmark datasets considered in @xcite ; namely, bless @xcite , leds @xcite , eval @xcite , and wbless @xcite . we do not consider the shwartz dataset @xcite , as the number of oov was high (38% for lexsub, retrofitting, and lear and 60% for counterfitting for glove). the evaluation is done by ranking the word pairs by cosine similarity and computing the mean average precision over the ranked list. the hypernymy directionality detection task is designed to detect which of the two terms is the hypernym of the other; that is, given two words w 1 and w 2 , is w 1 the hypernym of w 2 or vice versa. we consider two of the three datasets from @xcite ; namely, wbless and bibless @xcite . the classification setup is similar to @xcite and is done using the open source package provided by the authors. foot_1 ## results we now report on the results of our comparisons of lexsub to vanilla embeddings and baselines trained on the same lexical resource as lexsub. we use the main vector space embeddings in all our experiments except for hypernymy experiments, for which we use the hypernymy space embeddings. ## analysis in this section, we perform several analyses to understand the behaviors of our model and the baselines better, focusing on the following questions: q1: how well do lexsub's lexical subspaces capture the specific lexical relations for which they were optimized, as opposed to the other relations? q2: can the lexical subspaces and the manifolds in the main distributional space be exploited by a downstream neural network model? q3: how well do the models preserve relatedness in the main distributional space? to systematically quantify these results, we compute the mean average precision (map) over the top 100 neighbors for a list of query words. we use the words from the hyperlex @xcite and simlex @xcite datasets as the query words for this experiment. for each query word and for each lexical relation, we obtain a list of words from wordnet which are related to the query word through that particular lexical relation. these words form the gold-standard labels for computing the average precision for the query word. ## conclusion we presented lexsub, a novel framework for learning lexical subspaces in a distributional vector space. the proposed approach properly separates various lexical relations from the main distributional space, which leads to improved downstream task performance, interpretable learned subspaces, and preservation of distributional information in the distributional space. in future work, we plan to extend our framework to contextualized embeddings and expand the framework to support hyperbolic distances, which the ad-hoc distributional space. given a set of tokens in a corpus @xmath2, w 2 , . . . , w t ), we minimize the negative log likelihood function: training dataset. the ad-hoc model is trained on the wikitext-103 dataset @xcite . we preprocess the data by lowercasing all the tokens in the dataset across the splits, and limiting the vocabulary to top 100k words. ad-hoc lexsub model. the distributional component of our ad-hoc model is a two-layer qrnn-based language model @xcite with a 300-dimensional embedding layer and a 1,200-dimensional hidden layer. the batchsize, bptt length, and dropout ratio values for results we observe similar gains for ad-hoc lexsub on intrinsic evaluation in