Within-Between Lexical Relation Classification (2020)

Abstract: We propose the novel Within-Between Relation model for recognizing lexical-semantic relations between words. Our model integrates relational and distributional signals, forming an effective sub-space representation for each relation. We show that the proposed model is competitive and outperforms other baselines, across various benchmarks.

## introduction and related work recognizing lexical-semantic relations between words is beneficial for a variety of nlp tasks such as machine translation @xcite , relation extraction @xcite , natural language inference @xcite , and question answering @xcite . the lexical relation classification task assigns a word-pair (pair of words) to its corresponding relation out of a finite set of relations. this set contains lexical relations, including the random relation (indicating that the words are unrelated). two main lexical relation classification techniques are studied in the literature: path-based methods @xcite @xcite and distributional methods @xcite @xcite , with some effort for integrating the two @xcite . in this work we follow the distributional approach, which was shown to improve upon pathbased methods. this approach considers static word embeddings such as word2vec @xcite , glove @xcite ,and fast-text @xcite , which produce out-of-context vector representation for each word. note here that while contextualized embeddings @xcite have replaced the use of non-contextualized embeddings * equal contribution, order determined randomly. in many settings, static word embeddings remain the standard choice for lexical relation classification, since in this task the input word-pair is given out-of-context. taking the word embeddings as input, a classifier is trained while considering each word's representation in the pair. the recent spherere method @xcite , a purely distributional method that learns hyperspherical relation representation, presented state-of-the-art lexical relation classification results. while presenting state-of-the-art performance, prior distributional methods suffer from the "lexical memorization" problem @xcite . this problem arises when a test word-pair includes a rather frequent word in the training set, which is labeled by a dominant category in training. in such cases, the supervised model often ignores the second word in the input pair and resorts to the dominant training label according to the frequent word. notably, lexical memorization is common for prototypical hypernyms -"category" words that are frequently labeled as hypernyms. for example, the vast majority of training examples that include the word fruit are labeled as hypernymy (fruit is the hypernym of apple, banana, etc.). therefore, at inference time, the classifier is likely to predict the hypernymy relation even for unrelated word-pairs that contain fruit, e.g., (fruit,chair). another relevant line of research, which inspired our work, pertains to the integration of external lexical information to improve static word embeddings @xcite @xcite @xcite . most of these methods aim to modify the distributional vector space, originally learned from corpus co-occurrence data, by using additional relational constraints. to that end, these techniques rely on lexical databases, e.g., wordnet @xcite . notably, @xcite present the lexsub model and suggest training static word embeddings by integrating lexical-relation and distributional data, through the combination of two corresponding loss terms. when modeling lexical relations data, each relation is projected to a separate subspace. some of these ideas are adapted in certain parts of our work, while addressing the concrete goal of lexical relation classification rather than improving generic static word embeddings. in this work, we present the novel within-between relation (wbr) model, which is inspired both by previous relation classification models as well as by generic word embedding models that consider lexical relation constraints. this is performed through the combination of two objectives, both computed over the same projected sub-spaces, for each of the individual relations. specifically, our between objective aims to yield optimal classification of relation instances, while the within objective aims to bring pairs of words sharing a relation close to each other in the corresponding relation sub-space. these objectives allow the incorporation of both relation and negative sampling signals, altogether addressing much better the lexical memorization problem. ## the within-between relation model in this section, we present the wbr model. given a word-pair sharing a relation k, wbr is optimized to classify a word-pair to the correct relation (the between relation objective), and at the same time, separate it within the k relation space from other word-pairs that do not share the relation k. let i and k be vocabularies of words and relations, respectively. k contains lexical relations such as hypernym, antonym, etc., including the random relation (words are unrelated) and the co (stands for co-occurrence) relation that is shared by words that co-occur in the corpus. we further denote @xmath0. let u i , v @xmath1. we further denote @xmath2 let i @xmath3, j)|i k -→ j}, where i k -→ j means that words i and j share a directed relation k. in case of an undirected relation, it holds that (i k -→ j) ↔ (j k -→ i). note that in the specific case of the random relation, this assumption guarantees each word-pair (i, j) ∈ p is associated with a relation @xmath4. let f k : @xmath5. our goal is to learn parameters for f k s.t. the score f k ij is high if and only if (i, j) ∈ i k . in this work, we define f k ij α ## experimental setup and results in this section, we present the datasets, hyperparameters, and experiments that we conducted to evaluate our model and compare it with other methods. ## mitigating the lexical memorization problem the lexical memorization problem is alleviated by the introduction of the random relation. this feature plays a key role both for the between and within loss terms: given a word-pair (a, b) and their corresponding, ground truth relation r, we randomly sample a word n and associate the word-pair (a, n) with the random relation, unless r happens to be equal to the random relation beforehand (see algorithm 1). this unique mechanism is designed to balance each positive word-pair with a negative one, neutralizing the effect of multiple instances of the prototypical terms (e.g., animal, fruit, etc.) on the training objective, as a kind of regularization and data augmentation. for example, consider the positive data sample (animal, b). it will be balanced with a negative sample (animal, n). therefore, during the training phase, the between classifier learns to consider the random label each time it is given the hypernymy label. similarly, the corresponding within (hypernymy) classifier will encounter a negative sample for each positive sample. as a result, in the inference phase, the relation classifier does not always predict the hypernym relation for (animal, x) -the classifier will consider the features of x as well, and thus mitigates the lexical memorization problem. another way to ensure that each relation classifier exploits both words in the given pair is splitting them into two different linear projections' relation sets, ψ for the first word, and φ for the second. further, using the cosine similarity measure for computing f k ij , rather than a dot product, provides a normalization effect which neutralizes frequency biases, caused by typical larger norms for frequent words. ## conclusions we presented wbr -a novel model for lexical relation classification. wbr facilitates the novel between-within relation loss, enabling the exploitation of distributional information. wbr is evaluated on four different datasets, where it is shown to outperform various baselines across all evaluation metrics.