Learning Relation Representations from Word Representations
Abstract
Identifying the relations that connect words is an important step towards understanding human languages and is useful for various NLP tasks such as knowledge base completion and analogical reasoning.
Simple unsupervised operators such as vector oﬀset between two-word embeddings have shown to recover some speciﬁc relationships between those words, if any.
Despite this, how to accurately learn generic relation representations from word representations remains unclear.
We model relation representation as a supervised learning problem and learn parametrised operators that map pre-trained word embeddings to relation representations.
We propose a method for learning relation representations using a feed-forward neural network that performs relation prediction.
Our evaluations on two benchmark datasets reveal that the penultimate layer of the trained neural network-based relational predictor acts as a good representation for the relations between words.
1. Introduction
Diﬀerent types of relations exist between words in a language such as Hypernym , Meronym ,
Synonym etc.
Representing relations between words is important for various NLP tasks such as questions answering [Yang et al., 2017], knowledge base completion [Socher et al., 2013] and relational information retrieval [Duc et al., 2010].
Two main approaches have been proposed in the literature to represent relations between words.
In the ﬁrst approach, a pair of words is represented by a vector derived from a statistical analysis of a text corpus [Turney and Littman, 2005].
In a text corpus, a relationship between two words X and Y can be expressed using lexical patterns containing
X and Y as slot variables.
For example, “ X is a Y ” or “ Y such as X ” indicate that Y is a Hypernym of X
[Snow et al., 2005].
The elements of the vector representing the relation between two words correspond to the number of times those two words co-occur with a particular pattern in a corpus.
Given such a relation representation, the relational similarity between the relations that exist between the two words in two word-pairs can be measured by the cosine of the angle between the corresponding vectors.
We call this the holistic approach because a pair of words is treated as a whole rather than the two constituent words separately when creating a relation representation
[Turney, 2005].
Sparsity is a well- known problem for the holistic approach as two words have to co-occur enough in a corpus, or else no relation can be represented for rare or unseen word-pairs.
In contrast, the second approach for relation representation directly computes a relation representation from pre-trained word representations (i.e. word embeddings) using some relational operators .
Prediction-based word embedding learning methods [Pennington et al., 2014, Mikolov et al., 2013a] represent the meaning of individual words by dense, low-dimensional real-valued vectors by optimising diﬀerent language modelling objectives.
Although no explicit information is provided to the word embedding learning algorithms regarding the semantic relations that exist among words, prior work [Mikolov et al., 2013b] has shown that the learnt word embeddings encode remarkable structural properties pertaining to semantic relations.
They showed that the diﬀerence (vector oﬀset) between two word vectors (here-onwards denoted by PairDiff ) is an accurate method for solving analogical questions in the form “ a is to b as c is to ?”.
For example, king − man + woman results in a vector that is closest to the queen vector.
We call this approach compositional because the way in which the relation representation is composed by applying some linear algebraic relational operator on the the semantic representations of the the words that participate in a relation.
This interesting property of word embeddings sparked a renewed interest in methods that compose relation representations using word embeddings and besides PairDiﬀ, several other unsupervised methods have been proposed such as 3CosAdd and 3CosMult [Levy and Goldberg, 2014].
Despite the initial hype, recently, multiple independent works have raised concerns about of word embeddings capturing relational structural properties [Linzen, 2016, Schluter, 2018,
Liu et al., 2018, Rogers et al., 2017, Gladkova et al., 2016].
Although PairDiﬀ performs well on the Google analogy dataset, its performance for other relation types has been poor [Chen et al., 2017, Vylomova et al., 2016, K¨oper et al., 2015].
Vylomova et al.
[2016] tested for the generalisation ability of PairDiff using diﬀerent relation types and found that semantic relations are captured less accurately compared to syntactic relations.
Likewise, K¨oper et al.
[2015] showed that word embeddings are unable to detect paradigmatic relations such as Hypernym , Synonym and Antonyms .
Methods such as PairDiff are biased towards attributional similarities between individual words than relational similarities and fail in the presence of nearest neighbours.
We further discuss various limitations of the existing unsupervised relation composition methods in Section 2.2.
Considering the above-mentioned limitations of the unsupervised relation composition methods, a natural question that arises is whether it is possible to learn supervised relation composition methods to overcome those limitations.
In this paper, we model relation representation as learning a parametrised operator f
( a , b ; θ ) such that we can accurately represent the relation between two given words a and b from their word representations a and b , without modifying the input word embeddings.
For this purpose, we propose a Multi-class
Neural Network Penultimate Layer ( MnnPl ), a simple and eﬀective parametrised operator for computing relation representations from word representations.
Speciﬁcally, we train a nonlinear multilayer feed-forward neural network using a labelled dataset consisting of word-pairs for diﬀerent relation types, where the task is to predict the relation between two input words represented by their pre-trained word embeddings.
We ﬁnd that the penultimate layer of the trained neural network provides an accurate relation representation that generalises beyond the relations in the training dataset.
We emphasise that our focus here is not to classify a given pair to a relation in a pre-deﬁned set (relation classiﬁcation), but rather to obtain a good representation for the relation between the two words in the pair.
Our experimental results show that MnnPl signiﬁcantly outperforms unsupervised relational operators including PairDiff in two standard benchmark datasets, and generalises well to unseen out-of-domain relations.
2. Related Work
Relations between words can be classiﬁed into two types namely, contextual and lexical [Hendrickx et al., 2009, Nastase et al., 2013, G´abor et al., 2017].
Contextual relations are relations that exist between two words in a given speciﬁc context such as a sentence.
For example, given the sentence “the machine makes a lot of noise ”, a Cause-Effect relation exists between the machine and noise in this particular sentence.
More examples of Contextual relations can be found in Hendrickx et al.
On the other hand, Lexical relations hold between two words independent of the contexts in which those two words occur.
For instance, the lexical relation capital-of exists between London and England .
WordNet, for example, organises words into various lexical relations such as is-a-synonym-of , is-a-hypernym-of , is-an-meronym-of , etc.
Our focus in this paper is on representing lexical relations.
2.1 Relation Representation Operators
Word embeddings learning methods map words to real-valued vectors that represent the meanings of those words.
Given the embeddings of two words, Mikolov et al.
[2013b] showed that relations that hold between those words can be represented by the vector-oﬀset (diﬀerence) between the corresponding word embeddings.
This observation sparked a line of research on relational operators that can be used to discover relational information from word embeddings besides vector-oﬀset.
Using pre-trained word embeddings to represent relations is attractive for computational reasons.
Unlike holistic approaches that represent the relation between two words by lexico-syntactic patterns extracted from the co-occurrence contexts of the two words, relational operators do not require any co-occurrence contexts.
This is particularly attractive from a computational point of view because the number of possible pairings of n words grows O ( n 2 ), implying that we must retrieve co-occurrence contexts for all such pairings for extracting lexico-syntactic patterns for the purpose of representing the relations between words.
On the other hand, in the operational approach, once we have pre-trained the word embeddings we can compute the relation representations for any two words without having to re-learn anything.
For example, in applications such as relational search [Duc et al., 2011], we must represent the relation between two words contained in a user query.
Because we cannot anticipate all user queries and cannot precompute relation representations user queries oﬄine, relation compositional methods are attractive for relational search engines.
Operational methods of relation representation diﬀer from Knowledge Graph
Embedding (KGE) methods such as TransE [Bordes et al., 2013],
DistMult
[Yang et al., 2015],
CompIE [Trouillon et al., 2016], etc.
in the sense that in KGE, given a knowledge graph of tuples ( h, r, t ) in which a relation r relates the (head) entity h to the (tail) entity t , we must jointly learn embeddings for the entities as well as for the relations such that some scoring function is optimised.
For example, TransE scores a tuple ( h, t, r ) by the (cid:96) 1 or (cid:96) 2 norm of the vector ( h + r − t )
(we use bold fonts to denote vectors throughout the paper).
On the other hand, the relation composition problem that we consider in this paper does not attempt to learn entity embeddings or relation embeddings from scratch but use pre-trained word/entity embeddings to compose relation representations.
Therefore, operational methods of relation representation are attractive from a computational point of view because we no longer need to learn the word/entity embeddings and can focus only on the relation representation learning problem.
On the other hand, operational methods for relation representation diﬀer from those proposed for solving the analogy completion such as 3CosAdd [Mikolov et al., 2013b], 3CosMult [Levy and Goldberg, 2014], 3CosAvg and LRCos
[Drozd et al., 2016].
Analogy completion is the task of ﬁnding the missing word ( d ) in the two analogical word-pairs “ a is to b as c is to d ”.
To solve analogy completion, one must ﬁrst detect the relation in which the two words in the ﬁrst pair (i.e. ( a, b )) stand in, and then ﬁnd the word d that is related in the same way to c .
Methods that solve analogy questions typically consider the distances between the words of the two pairs in some common vector space.
For example, 3CosAdd computes the inner product between the vector ( b − a + c ) and the word embedding d for each word in the vocabulary.
If the vectors are l 2 normalised then inner-product is equivalent to cosine similarity, which can be seen as a calculation involving cosine similarity scores for three pairs of words ( b , d ) , ( a , d ) and ( c , d ) explaining its name 3CosAdd.
3CosMult, on the other hand, considers the same three cosine similarity scores but in a multiplicative formula.
However, analogy completion methods such as 3CosAdd or 3CosMult cannot be considered as relation representation methods because they do not create a representation for the relation between a and b at any stage during the computation.
Hakami and Bollegala [2017a] compared diﬀerent unsupervised relational operators such as PairDiff , concatenation, vector addition and elementwise multiplication and reported that PairDiff to be the best operator for analogy completion whereas, elementwise multiplication was the best for link prediction in knowledge graphs.
A recent work [Hakami et al., 2018] has theoretically proven that PairDiff to be the best linear unsupervised operator for relation representation when the relational distance (similarity) between two word-pairs is measured in term of the squared Euclidean distance between the corresponding relation representation vectors.
2.2 Limitations of Unsupervised Relational Operators
Recently, several limitations have been reported of the existing unsupervised relational representation operators [Linzen, 2016, Rogers et al., 2017].
In particular, the distance between word embeddings in a semantic space signiﬁcantly aﬀects the performance of PairDiff in analogy completion.
Speciﬁcally, to measure the relational similarity between ( a, b ) and ( c, d ) pairs using PairDiff , prior work compute the inner-product between the normalised oﬀset vectors: ( a − b ) (cid:62) ( c − d ).
This is problematic because the task of measuring relational similarity between the two word-pairs is simply decomposed into a task of measuring lexical similarities between individual words of the pairs.
Speciﬁcally, the above inner-product can be rewritten as a (cid:62) c − a (cid:62) d − b (cid:62) c
+ b (cid:62) d .
This value can become large, for example when a is highly similar to c or b is highly similar to d , irrespective of the relationship between a and b , and c and d .
As a concrete example of this issue, consider measuring the relational similarity between ( water , riverbed ) and each of the two word-pairs ( traﬃc , street ) and ( water , drink )).
In this case, water ﬂows-In riverbed is the implicit relation expressed by the two words in the stem word-pair ( water , riverbed ).
Therefore, the candidate pair ( traﬃc , street ) is relationally more similar to the stem word-pair than ( water , drink ) because ﬂows-In also holds between traﬃc and street .
However, if we use pre-trained GloVe word embeddings [Pennington et al., 2014] with PairDiff as the relation representation, then ( water , drink ) reports a higher relational similarity score (0 . 62) compared to that for ( traﬃc , street ) (0 . 42) because of the lexical similarities between the individual words.
PairDiﬀ was originally evaluated by Mikolov et al.
[2013b] using semantic and syntactic relations in the Google dataset such as Capital-City , Male-Female , Currency , City-in-State , singular-plural , etc.
However, more recent works have shown that although PairDiﬀ can accurately represent the relation types in the Google dataset, it fails on other types of relations [K¨oper et al., 2015, Fu et al., 2014].
For example, K¨oper et al.
[2015] showed PairDiff cannot detect paradigmatic relations such as hypernymy, synonymy and antonymy, whereas
Fu et al.
[2014] reported that hypernym-hyponym relation is more complicated and a single oﬀset vector cannot completely represent it.
The space of unsupervised operators proposed so far in the literature is limited in the sense that the operators pre-deﬁned and ﬁxed, and cannot be adjusted to capture the actual relations that exist between words.
It is unrealistic to assume that the same operator can represent all relation types from the word embeddings learnt from diﬀerent word embedding learning algorithms.
On the other hand, there are many datasets such as SemEval
Task2, Google, MSR, SAT verbal analogy questions etc., which already provide examples of the types of relations that actually exist between words.
Our proposed supervised relational composition method learns a parametrised operator implemented as a neural network, which can be trained to better represent relations between words.
2.3 Relation Detection using Word Embeddings
Word embeddings have been used as features in prior work for learning lexical relations between words.
Given two words, Vylomova et al.
ﬁrst represent the relation between those words using PairDiff and then train a multi-class classiﬁer for classifying diﬀerent relation types.
Methods that focus on detecting a particular type of relation between two words such as hypernymy , have also used unsupervised relation composition operators such as PairDiff to create a feature vector for a word-pair [Carmona and Riedel, 2017, Levy et al., 2015b, Roller et al., 2014, Fu et al., 2014].
Carmona and Riedel [2017] and Roller et al.
[2014] train respectively a logistic regression classiﬁer and a linear support vector classiﬁer using word-pairs represented by the PairDiff or concatenation of the corresponding pretrained word embeddings.
Levy et al.
[2015b] used PairDiff and vector concatenation as operators for representing the relation between two words and evaluated the representations in a lexical entailment task and a hypernym prediction task.
They found that the representations produced by these operators did not capture relational properties but simply retained the information in individual words, which was then used by the classiﬁers to make the predictions.
Similarly, Fu et al.
[2014] observe that PairDiff is inadequate to induce the hypernym relation.
Chen et al.
[2017] analysed PairDiff on a number of diﬀerent relation types and found that its performance varies signiﬁcantly across relations.
3. Supervised Relational Operators
Our goal in this paper is to learn a parametrised two-argument function f ( · , · ; θ ) that can accurately represent the relation between two given words a and b using their pre-trained d -dimensional word embeddings a , b ∈ R d .
Here, θ denotes the set of parameters that governs the behaviour of f , which can be seen as a supervised operator that outputs a relation representation from two input word representations.
The output of f , for example, could be a vector that exists in the same or a diﬀerent vector space as a and b , as given by (1).
[Equation]
In general d (cid:54) = m and word and relation representations can have diﬀerent dimensionalities and even when d = m they might be in diﬀerent vector spaces.
We could extend this deﬁnition to include higher-order relation representations such as matrices or tensors but doing so would increase the computational overhead.
Therefore, we limit supervised relational operators such that they return vectors as given by (1) in this paper.
We note that unsupervised relational operators such as PairDiff and vector concatenation are speciﬁc instances of this deﬁnition.
For example, for PairDiff we have f ( a , b ; θ ) =
a − b ( m = d ) , and for vector concatenation we have f ( a , b ; θ ) =
a ⊕ b ( m = 2 d ), where ⊕ denotes concatenation of two vectors.
In unsupervised operators, θ is a constant that does not inﬂuence the output relation embedding.
3.1 Relation Representation via Relation Prediction
We implement the proposed supervised relation composition operator as a feed-forward neural network with one or more hidden layers followed by a softmax layer as shown in
Figure 1.
Weight matrices for the hidden layers are W 1 and W 2 , whereas the biases are s 1 and s 2 .
g refers to the nonlinear activation for the hidden layers.
We experiment with diﬀerent nonlinearities in the hidden layers.
Using a dataset D
= { ( a i , r i , b i ) } Ni = of word-pairs ( a i , b i ) with relations
r i , we train the neural network to predict r i given the concatenated pre-trained word embeddings
a
i ⊕ b i as the input.
We minimise the (cid:96)regularised cross-entropy loss over the training instances.
After training the neural network, we use its penultimate layer (i.e. the output of the ﬁnal hidden layer) as the relation representation for a word-pair.
We call this method Multi-class Neural Network Penultimate
Layer ( MnnPl ).
We emphasise that our goal is not to classify a given pair into a speciﬁc set of relations, but rather to ﬁnd a representation of the relation between any pair of words.
Therefore, we test the learnt relation representation using relations that are not seen during training (i.e. out-of-domain examples) by holding out a subset of relations during training.
4. Experiments and Results
We evaluate the relation embeddings learnt by the proposed MnnPl on two standard tasks: out-of-domain relation prediction and measuring the degree of relational similarities between two word-pairs.
In Section 4.1, we ﬁrst introduce the relational training datasets and the input word embedding models that we used to compose relation embeddings.
Next, in Section 4.2, we describe the experimental setup that we follow to train the proposed method.
We compare the performance of the MnnPl with various baseline methods as illustrated in Section 4.3.
In Section 4.4 and 4.5, we discuss the experiments conducted on the out-of-domain and in-domain relation prediction task, respectively.
The task of measuring the degree of relational similarities is presented in Section 4.6.
In short, each two word-pairs in the dataset for this task has a manually assigned relational similarity score, which we consider as the gold standard rating for relational similarity.
4.1 Training Dataset and Word Embeddings
We used two previously proposed datasets for evaluating MnnPl : BATS 1 [Gladkova et al., 2016] and DiﬀVec
[Vylomova et al., 2016].
BATS is a balanced dataset that containsmain relation types, two are semantic relations (Lexicographic and Encyclopaedic) and the other two are syntactic relations (Inﬂectional and Derivational).
Each main category hasdiﬀerent sub-relation types and 50 word-pairs are provided for each relation (2,000 unique word-pairs in total).
DiﬀVec covers 36 subcategories that are classiﬁed into 15 main relation types in total (31 semantic and 6 syntactic).
The dataset is unbalanced because a diﬀerent number of word-pair examples assigned to each relation, in total it has 12,452 word-pairs.
We exclude relations that has less than 10 examples from experiments.
For word embeddings, we use CBOW, Skip-Gram (SG)
[Mikolov et al., 2013a] and
GloVe [Pennington et al., 2014] as the input to the proposed method.
For consistency of the comparison, we train all word embedding learning methods on the ukWaC corpus [Ferraresi et al., 2008], a web-derived corpus of English consisting of ca.
2 billion words.
Words that appear less than 6 times in the entire corpus are truncated, resulting in a vocabulary of 1,371,950 unique words.
We use the publicly available implementations by the original authors for training the word embeddings using the recommended parameters settings.
Speciﬁcally, GloVe model was trained with window size 15, 50 iterations, weighting function parameters x max = 100 , α = 0 .
CBOW and SG embeddings were trained with window size 8, 25 negative samples, 15 iterations, sampling parameter equal to 10 − 4 .
In addition to the prediction-based word embeddings created using CBOW and GloVe, we use Latent Semantic Analysis (LSA) to obtain counting-based word embeddings [Deerwester et al., 1990, Turney and Pantel, 2010, Clark, 2015].
A co-occurrence matrix M ∈
R n × n is ﬁrst constructed considering the 50k most frequent words in the corpus to avoid data sparseness.
The raw counts are weighted following positive point-wise mutual information (PPMI) method.
Subsequently, singular value decomposition (SVD) is applied to reduce the dimensionality M to lower rank matrices U k S k V k
(cid:62) , where S k is a diagonal matrix that has the largest k singular values of M as the diagonal elements.
U k and V k are orthogonal matrices of singular vectors of the corresponding k singular values.
Following Levy et al.
[2015a], S k is ignored when representing the words (i.e. M = U k ).
4.2 Training Details
We use the word embeddings trained on the ukWaC with 50 dimensions as the input to the neural network.
Overall, we found (cid:96) 2 normalisation of word embeddings to improve results.
We use Stochastic Gradient Descent (SGD) with Momentum [Qian, 1999] with mini-batch size of 128 to minimise the (cid:96) 2 regularised cross-entropy error.
All parameters are initialised by uniformly sampling from [ − 1 , +1] and the initial learning rate is set to 0.1.
Dropout regularisation is applied with a 0.25 rate.
Tensorﬂow is used to implement the model.
We train the models till the convergence on a validation split.
We used the Scholastic Aptitude
Test (SAT) 374 multiple choice analogy questions dataset [Turney et al., 2003] for validating the hyperparameter values.
Speciﬁcally, we selected the number of the hidden layers among { 1, 2, 3 } and the activation function g of the hidden layers among { tanh, relu, linear } .
On the validation dataset, we found the optimal conﬁguration was to set the number of hidden layers to two and the nonlinear activation to tanh.
The optimal (cid:96) 2 regularisation coeﬃcient λ was 0 .
We train the models till the convergence on the validation dataset.
These settings performed consistently well in all our evaluations.
4.3 Baseline Methods
We compare the relation representations produced by MnnPl against several baselines as detailed next.
Note that the considered baselines produce relation representations for word-pairs.
Unsupervised Baselines:
We implement the following unsupervised relational operators for creating relation representations using word embeddings Hakami and Bollegala [2017b]: PairDiff , Concatenation ( Concat ), elementwise addition ( Add ) and elementwise multiplication ( Mult ).
These operators are unsupervised in the sense that there are no parameters in those operators that can be learnt from the training data.
Supervised Baselines
We design a supervised version of the Concat operator parametrised by a weight matrix W ∈ R d × m and a bias vector s ∈ R m to compute a relation representation r for two words a and b as given in (2).
[Equation]
We call this baseline as the Supervised Concatenation ( Super-Concat ).
Likewise, we design a supervised version of PairDiff , which we name Super-Diﬀ as follows:
[Equation]
In addition to the above supervised operators, we use the bilinear operator proposed by [Hakami et al., 2018] (given in (4)) as a supervised relation representation method.
[Equation]
Here, A ∈ R d × d × m is a 3-way tensor in which each slice is a d × d real matrix.
The ﬁrst term in (4) corresponds to the pairwise interactions between a and b .
P , Q ∈ R d × d are the projection matrices involving ﬁrst-order contributions respectively of a and b towards r .
We refer to this operator as BiLin .
We train the above-mentioned three supervised relational operators using a marginbased rank loss objective.
Speciﬁcally, we minimise the distance between the relation representations of the analogous pairs (positive instances), while maximising the distance between the representations of non-analogous examples (negative instances) created via random perturbations.
Given a set of word pairs S r that are related by the same relation, we generate positive training instances (( a, b ) , ( c, d )) by pairing word-pairs ( a, b ) ∈ S
r and ( c, d ) ∈ S r .
Next, to generate negative training instances, we corrupt a positive instance by pairing ( a, b ) ∈ S r with a word-pair ( c (cid:48) , d (cid:48) )
∈ S r (cid:48) that belongs to a diﬀerent relation r (cid:48) (cid:54) = r .
One negative instance is generated for each analogous example in our experiments, resulting in a balanced binary labelled dataset.
The regularised training objective L ( D ; θ ) is given by (5).
[Equation]
Here, δ is a margin hyperparameter set to 1 according to the best accuracy on the SAT validation dataset.
The regularisation coeﬃcient λ is set separately for each parameter in the diﬀerent supervised relation composition operators using the SAT validation dataset.
For SuperConcat and Super-Diﬀ , regularising W and s resulted in lowering the accuracy on SAT questions.
Therefore, no regularisation is applied for those two operators.
For the BLin operator, the best regularisation coeﬃcient for the tensor A on the validation dataset was 0.1.
However, regularising P and Q decreased the performance on the validation set, and therefore were not regularised.
4.4 Out-of-domain Relation Prediction
A critical evaluation criterion for a relation representation learning method is whether it can accurately represent not only the relations that exist in the training data that was used to learn the relation representation but can also generalise to unseen relations ( out-of-domain) .
Therefore, to evaluate the diﬀerent relation representation methods, we employ them in an out-of-domain relation prediction task.
Speciﬁcally, we use diﬀerent relations for testing than that used in training.
No training is required for unsupervised operators.
Next, we describe the evaluation protocol in detail.
Lets denote a set of relation types by R and a set of word-pairs covering the relations in R by D .
First, we randomly sample ﬁve target relations from the dataset to construct a relation set R t for testing and the remainder represents a set of source relations R s that is used for training the supervised relational operators including the supervised baselines and the proposed MnnPl .
We use the set D s of word-pair instances covering R s to learn the supervised operators by predicting the relations in R s .
To evaluate the performance of such operators, we use the relational instances in the test split D t that cover the out-of-domain relations in R t .
We conduct 1-NN relation classiﬁcation on D t dataset.
The task is to predict the relation that exists between two words a and b from the sampled relations in R t .
Speciﬁcally, we represent the relation between two words using each relational operator on the corresponding word embeddings.
Next, we measure the cosine similarity between representations for the stem pair and all the word-pairs in D t .
For each target word-pair, if the top-ranked word-pair has the same relation as the stem pair, then it is considered to be a correct match.
Note that we do not use D t for learning or updating the (supervised) relational operator but use it only for the 1-NN relation predictor.
We repeat this process ten times by selecting diﬀerent R s and
R t relation sets and use leave-one-out evaluation for the 1-NN as the evaluation criteria.
We compute the (micro-averaged) classiﬁcation accuracy of the test sets as the evaluation measure.
Because each relation type in an out-of-domain relation set has multiple relational instances, a suitable relation representation method retrieves the related pairs for a target pair at the top of the ranked list.
For this purpose, we measure Mean Average Precision (MAP) for the relation representation methods.
To derive further insights into the relation representations learnt, following Nastase et al.
[2013], we use the notion of “near” vs. “far” analogies considering the similarities between the corresponding words in the two related pairs.
For example, ( tiger , feline ), ( cat , animal ) and ( motorcycle , vehicle ) are all instances of the is-a-hypernym-of relation.
One could see that ( tiger , feline ) is closer to ( cat , animal ) than ( motorcycle , vehicle ).
Here, tiger and cat are similar because they are both animals; also feline and animal have shared attributes.
On the other hand, the corresponding words in the two pairs ( tiger , feline ) and ( motorcycle , vehicle ) have low attributional similarities between tiger and motorcycle or between feline and vehicle .
Detecting near analogies using word embeddings is easier compared to far analogies because attributional similarity can be measured accurately using word embeddings.
For this reason, we evaluate the accuracy of a relation representation method at diﬀerent degrees of the analogy as follows.
Given two word-pairs, we compute the cross-pair attributional similarity using SimScore deﬁned by (6).
[Equation]
Here, sim( x , y ) is the cosine similarity between x and y .
Next, we sort the word-pairs in the descending order of their SimScores (i [Table] e. from near to far analogies).
Examples of far and near analogies with SimScores for some selected word-pairs are presented in Table
[Table] To alleviate the eﬀect of attributional similarity between two word-pairs in our evaluation [Table] we remove the 25% top-ranked (nearest) pairs for each stem pair.
Consequently, a relation representation method that relying only on attributional similarity is unlikely to accurately represent the relations between words
[Table]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Table]
The average accuracy (Acc) and the MAP of the relation representation operators for
CBOW, SG, GloVe and LSA embeddings are presented in Table 2.
As can be observed among the diﬀerent embedding types, MnnPl consistently outperforms all other methods in both Acc and MAP score.
The diﬀerences between MnnPl and other methods for all rounds and target relations are statistically signiﬁcant ( p < 0 .
according to a paired t-tes.
CBOW embeddings report the best Acc and MAP scores for the two datasets in contrast to all other embedding models.
We also assess how good such relational operators are on the in-domain relation prediction task, wherein the task is to represent relational instances that belong to the relation set used on training the models.
We ﬁnd that MnnPl can accurately represent relations in this in-domain setting as well (see Section 4.5).
To further evaluate the accuracy of the diﬀerent relational operators on diﬀerent relation types, we break down the evaluation per major semantic relation type in the BATS dataset as shown in Table 3.
We see that lexicographic relations are more diﬃcult compared to encyclopaediac relations for all methods.
Overall, the proposed MnnPl consistently outperforms other methods for both types of semantic relations.
On the other hand, PairDiff performs signiﬁcantly worse for lexicographic relations.
We believe that this result explains
PairDiff ’s superior performance on the Google analogy dataset, which contains a large proportion of encyclopaediac relations such as capital-common-countries , capital-currency , city-in-state , and family .
ADD achieves the second best accuracy for Encyclopedic relations (where PairDiff is only slightly behind it), whereas Concat follows MnnPl in term of MAP scores.
For encyclopaediac relations, the head words can be grouped into a sub-space in the embedding space that is roughly aligned with the sub-space of the tail words [Liu et al., 2018, Bouraoui et al., 2018].
For instance, in the country-capital relation the head words represent countries while the tail words represent cities.
On the other hand, lexicographic relation types do not have speciﬁc sub-spaces for the related head and tail words, which means that the oﬀset vectors would not be suﬃciently parallel for PairDiff to work well.
This is further evident from Figure 2 where the average cosine similarity scores between the relation embeddings computed using PairDiff is signiﬁcantly smaller for the lexicographic relations compared to that for the encyclopaediac relations on the
BATS dataset.
Consequently, the performance of PairDiff on lexicographic relations is poor, whereas MnnPl reports the best results.
As mentioned in Section 2.2, PairDiff is biased towards the attributional similarity between words in two word-pairs compared.
To evaluate the eﬀect of this, we group test cases in the DiﬀVec dataset into two categories: (a) lexical-overlap (i.e. there is exactly one word in common between the two word-pairs in a test case) and (b) lexical-nonoverlap (i.e. no words are common between the two word-pairs in a test case).
In other words, given the test word-pair ( a, b ), then if there is a train word-pair ( a, c ), ( b, c ), ( c, a ) or ( c, b ) we consider this case as a lexical-overlap between the test and the train datasets.
For example, ( animal , cat ) and ( animal , dog ) has lexical-overlap because animal is a common word in the two pairs.
Figure 3 shows the average 1-NN classiﬁcation accuracy for the best unsupervised operator PairDiff and MnnPl .
We see that the performance drops signiﬁcantly from lexical-overlap to lexical-nonoveralp by ca.
10% for PairDiff , whereas that drop is ca.
1.8% for MnnPl .
This result indicates that MnnPl is aﬀected less by attributional similarity compared to PairDiff .
4.5 In-domain Performance of the Relation Representation Methods
We evaluate the performance of the relation representation operators considering in-domain setting, wherein we test the performance on relational instances belong to relation types used in the training set.
Recall that R and D refer to the set of relations and the set of relational instances covering such relations, respectively.
In the in-domain setting, we do not need to split R to source and target relation sets.
Instead, we implement 5-stratiﬁed folds cross-validation considering the set of relational instances in the dataset D
We use 1NN and MAP as we did in the out-of-domain experiment.
So in-domain experiment setting is very similar to out-of-domain experiment expect in the latter we use R s (cid:54)
=
R t for the evaluation.
Detailed results for in-domain evaluation are presented in Table 4.
4.6 Measuring the Degree of Relational Similarity
The relational similarity is the correspondence between the relations of two word-pairs.
To measure a relational similarity score between two pairs of words, one must ﬁrst identify the relation in each pair to perform such comparison.
Suitable relation embeddings should highly correlate with human judgments of relational similarity between word-pairs.
For this
[Table]
task, we use the dataset proposed by Chen et al.
[2017] 3 which is inspired by SemEvaltask 2 dataset [Jurgens et al., 2012].
In this dataset, humans are asked to score pairs of words directly focusing on a comparison between instances with similar relations.
For examples, in Location:Item relation, the pairs ( cupboard , dishes ) and ( kitchen , food ) are assigned higher relational similarity score (6.18) than the pairs ( cupboard , dishes ) and ( water , ocean ) which is rated 3.8.
Instances of this relation (X, Y) can be expressed by multiple patterns such as “X holds Y” or “Y in the X”, and one reason that the second example is assigned low score is that the words in the pair ( water , ocean ) are ordered reversely compared to other pairs.
Chen et al.
[2017] dataset consist of 6,194 word-pairs across 20 semantic relation subtypes.
We calculated the relational similarity score of two pairs as the cosine similarity between the corresponding relation vectors generated by the considered operators.
Then, we measure the Pearson correlation coeﬃcient between the average human relational similarity ratings and the predicted scores by the methods.
For this task, we choose to train the supervised methods on BATS as the overlap of the relation set between BATS and Chen datasets are small.
We exclude any word-pairs in Chen dataset that appears in the training data.
Table 5 shows Pearson correlations for all the four embedding models and the relational representation methods across all relations, where high values indicate a better agreement with the human notion of relational similarity.
As can be observed, the proposed MnnPl correlated better with human ratings than the supervised and unsupervised baselines.
According to the Fisher transformation test of statistical signiﬁcant, the reported correlations of MnnPl is statistically signiﬁcant at the 0.05 signiﬁcant level.
Interestingly, the
Concat baseline shows a stronger correlation coeﬃcient than PairDiff .
Moreover, for
SG and LSA embeddings, Add and Mult are considered stronger than PairDiff .
In consistent with out-of-domain relation prediction task, CBOW embedding perform better than other embeddings for measuring the degree of relational similarity.
Indeed, measuring the degree of relational similarity is a challenging task and required qualiﬁed ﬁne-grained relation embeddings to obtain accurate scores of relational instances.
[Table]
5. Conclusion
We considered the problem of learning relation embeddings from word embeddings using parametrised operators that can be learnt from relation-labelled word-pairs.
We experimentally showed that the penultimate layer of a feed-forward neural network trained for classifying relation types ( MnnPl ) can accurately represent relations between two given words.
In particular, some of the disﬂuencies of the popular PairDiff operator can be avoided by using MnnPl , which works consistently well for both lexicographic and encyclopaedic relations.
The relation representations learnt by MnnPl generalise well to previously unseen (out-of-domain) relations as well, even though the number of training instances is typically small for this purpose.
Our analysis highlighted some important limitations in the evaluation protocol used in prior work for relation composition operators.
Our work questions the belief that unsupervised operators such as vector oﬀset can discover rich relational structures in the word embedding space.
More importantly we show that simple supervised relational composition operators can accurately recover the relational regularities hidden inside word embedding spaces.
We hope our work will inspire the NLP community to explore more sophisticated supervised operators to extract useful information from word embeddings in the future.
Recently, Roller et al.
[2018] show that accessing lexical relations such as hypernym relying only on distributional word embeddings that are trained considering 2-ways cooccurrences between words is insuﬃcient.
They illustrate the advantages of using the holistic (pattern-based) to detect such relations.
Indeed, it is expected that the holistic and the compositional approaches for representing relations have complementary properties since the holistic uses lexical contexts in which the two words of interest co-occur, while the compositional uses only their embeddings [Shwartz et al., 2016].
Interesting future work includes unifying the two approaches for relation representations