C ONTEXT M OVER ’ S D ISTANCE & B ARYCENTERS : O PTIMAL TRANSPORT OF CONTEXTS FOR BUILDING REPRESENTATIONS
Anonymous authors
Paper under double-blind review
A BSTRACT
We propose a uniﬁed framework for building unsupervised representations of entities and their compositions, by viewing each entity as a histogram (or distribution) over its contexts.
This enables us to take advantage of optimal transport and construct representations that effectively harness the geometry of the underlying space containing the contexts.
Our method captures uncertainty via modelling the entities as distributions and simultaneously provides interpretability with the optimal transport map, hence giving a novel perspective for building rich and powerful feature representations.
As a guiding example, we formulate unsupervised representations for text, and demonstrate it on tasks such as sentence similarity and word entailment detection.
Empirical results show strong advantages gained through the proposed framework.
This approach can potentially be used for any unsupervised or supervised problem (on text or other modalities) with a co-occurrence structure, such as any sequence data.
The key tools at the core of this framework are Wasserstein distances and Wasserstein barycenters, hence raising the question from our title.
I NTRODUCTION
One of the main driving factors behind the recent successes in machine learning and natural language processing has been the development of better representation methods for data modalities.
Examples include continuous vector representations for language (Mikolov et al., 2013; Pennington et al., 2014),
Convolutional Neural Network (CNN) based text representations (Collobert & Weston, 2008; Kim, 2014; Kalchbrenner et al., 2014; Severyn & Moschitti, 2015; Deriu et al., 2017), or via other neural architectures such as RNNs, LSTMs (Hochreiter & Schmidhuber, 1997; Kiros et al., 2015), all sharing one central idea – to map input entities to dense vector embeddings lying in a low-dimensional latent space where the semantics of the inputs are preserved.
While these existing methods represent each entity of interest (e.g., a word) directly as a single point in space (i.e., its embedding vector), we here propose a fundamentally different approach.
We focus on co-occurrence information of the entities and their contexts (e.g. context words), and leverage embeddings of the contexts instead of the original entities.
So instead of a single point per entity, we represent each entity by the histogram of its contexts , where the contexts themselves are represented as points in a suitable metric space.
This allows us to cast the distance between histograms associated with the contexts as an instance of the optimal transport problem (Monge, 1781; Kantorovich, 1942;
Villani, 2008).
Our resulting framework then intuitively seeks to minimize the cost of moving the contexts of a given entity to the contexts of another, which motivates the naming Context Mover’s Distance (CMD).
Note that the contexts here can be words, phrases, sentences, or any generic entities co-occurring with our entities to be represented, and these entities further could be of various kinds, including e.g., products such as movies or web-advertisements (Grbovic et al., 2015), nodes in a graph (Grover & Leskovec, 2016), sequence data, or any other entities (Wu et al., 2017).
Any co-occurrence structure will allow construction of the histogram information, which is the crucial building block of our approach.
The main motivation for our proposed approach here comes from the domain of natural language, where the entities (words, phrases, or sentences) generally have different semantics depending on the context under which they are present.
Hence, it is important that we consider representations that are able to effectively capture such inherent uncertainty and polysemy, and we will argue that histograms (or probability distributions) over embeddings allows to capture more of this information compared to point-wise embeddings alone.
We will call this histogram over contexts embeddings as the distributional estimate of our entity of interest, while we refer to the individual embeddings of contexts as point estimates .
The connection to optimal transport at the level of entities and contexts paves the way to make better use of its vast toolkit (like Wasserstein distances, barycenters, barycentric coordinates, etc.) for applications in NLP, which in the past has primarily been restricted to document distances of original words (Kusner et al., 2015; Huang et al., 2016), as opposed to contexts.
Thanks to the entropic regularization introduced by Cuturi (2013), optimal transport computations can be carried out efﬁciently in a parallel and batched manner on GPUs.
Contributions:
• Employing the notion of optimal transport of contexts as a distance measure, we illustrate how our framework can be of beneﬁt for various important tasks, including word and sentence representations, sentence similarity, as well as hypernymy (entailment) detection.
The method is static and does not require any additional learning, and can be readily used on top of existing embedding methods.
• The resulting representations via the transport map give a clear interpretation of the resulting distance (see also Figure 1), on top of the co-occurrence information.
• Our Context Mover’s Distance can be used to measure any kind of distance (even asymmetric) between words, by deﬁning a suitable underlying cost on the movement of contexts, which we show can lead to a state-of-the-art metric for word entailment.
• Deﬁning the transport over contexts has the added beneﬁt that the representations are compositional - they directly extend from entities to groups of entities (of any size), such as from word to sentence representations.
To this end, we utilize the notion of Wasserstein barycenters, which to the best of our knowledge has never been considered in the past.
The proposed framework is not speciﬁc to words or sentences but can allow building unsupervised representations for any entity and composition of entities, where a co-occurrence structure can be devised between entities and their contexts, and we leave this direction for a future work.
R ELATED W ORK
Most of the previous work in building representations for natural language has been focused towards vector space models, in particular, popularized through the groundbreaking work in
Word2vec (Mikolov et al., 2013) and GloVe (Pennington et al., 2014).
The key idea in these models has been to map words which are similar in meaning to nearby points in a latent space.
Based on which, many works (Levy & Goldberg, 2014a; Melamud et al., 2015; Bojanowski et al., 2016) have suggested specializing the embeddings to capture some particular information required for the task at hand.
One of the problems that still persists is the inability to capture, within just a point embedding, the various semantics and uncertainties associated with the occurrence of a particular word (Huang et al., 2012; Guo et al., 2014).
A recent line of work has proposed the view to represent words with Gaussian distributions or mixtures of Gaussian distributions (Vilnis & McCallum, 2014; Athiwaratkun & Wilson, 2017), or hyperbolic cones (Ganea et al., 2018) for this purpose.
Also, concurrent works by Muzellec & Cuturi (2018) and Sun et al.
(2018) have suggested using elliptical and Gaussian distributions endowed with a Wasserstein metric respectively.
While these already provide richer information than typical vector embeddings, their form restricts what could be gained by allowing for arbitrary distributions.
In addition, hyperbolic embeddings (Nickel & Kiela, 2017; Ganea et al., 2018) are so far restricted to supervised tasks (and even elliptical embeddings (Muzellec & Cuturi, 2018) to the most extent), not allowing unsupervised representation learning as in the focus of the paper here.
To this end, we propose to represent each word with a distributional estimate (i.e., histogram over context embeddings), that inherently relies upon the empirically obtained co-occurrence information of a word and its contexts.
Hence, this naturally allows for the use of optimal transport (or Wasserstein metric) in the space containing the contexts, and leads to an interpretation (Figure 1) which is not available in the above approaches.
Amongst the few explorations of optimal transport in NLP, i.e., document distances (Kusner et al., 2015; Huang et al., 2016), topic modelling (Rolet et al., 2016; Xu et al., 2018), document clustering (Ye et al., 2017), bilingual lexicon induction (Zhang et al., 2017), or orthogonal Procrustes mapping (Grave et al., 2018), the focus has been on transporting words directly.
For example, the
Word Mover’s Distance (Kusner et al., 2015) casts ﬁnding the distance between documents as an optimal transport problem between their bag of words representation.
Our approach is different as we consider the transport over contexts instead, and use it to propose a representation for words or entities.
This provides the added ﬂexibility to establish any kind of distance between entities and extend the representation to composition of entities in a principled manner, as will be illustrated further through the examples of entailment detection and sentence representation respectively.
B ACKGROUND ON O PTIMAL T RANSPORT
Optimal Transport (OT) provides a way to compare two probability distributions deﬁned over a space G (commonly known as the ground space), given an underlying distance or more generally a cost of moving one point to another in the ground space.
In other terms, it lifts a distance between points to a distance between distributions.
Other methods of comparing distributions, such as Kullback-Liebler (KL), squared Hellinger, etc., only focus on the probability mass values, thus ignoring the geometry of the ground space: something which we utilize throughout this work via OT.
Also, some divergences like KL are not deﬁned when the supports of distributions under comparison don’t match.
Hence, we give a short yet formal background on OT in the discrete case.
Linear Program Formulation.
Consider an empirical probability measure of the form µ = (cid:80) ni
=1 a i δ ( x i ) where
X =
( x 1 , . .
, x n ) ∈ G n , δ ( x ) denotes the Dirac (unit mass) distribution at point x ∈ G , and ( a 1 , . .
. , a n ) lives in the probability simplex Σ n := (cid:8) p ∈ R n + | (cid:80)
[Equation]
1 p i
[Equation] [Equation]
Now given a second empirical measure, ν = (cid:80) mj
=1 b j δ ( y j ) , with Y =
( y 1 , . . .
, y m ) ∈ G m , and ( b 1 , . . .
, b m ) ∈ Σ m , and if the ground cost of moving from point x i to y j is denoted by M ij , then the
Optimal Transport distance between µ and ν is the solution to the following linear program.
[Equation]
Here, the optimal T ∈ R n × m is referred to as the transportation matrix : T ij denotes the optimal amount of mass to move from point x i to point y i .
Intuitively, OT is concerned with the problem of moving goods from factories to shops in such a way that all the demands are satisﬁed and the overall transportation cost is minimal.
Distance.
When G = R d and the cost is deﬁned with respect to a metric D G over G (i.e., M ij =
D G ( x i , y j ) p for any i, j ), OT deﬁnes a distance between empirical probability distributions.
This is the p -Wasserstein distance, deﬁned as W p ( µ, ν ) :
=
OT ( µ, ν ; D p G ) 1 /p .
In most cases, we are only concerned with the case where p = 1 or 2 .
The cost of exactly solving OT problem scales at least in O ( n 3 log( n ))
( n being the cardinality of the support of the empirical measure) when using network simplex or interior point methods.
Following
Cuturi (2013), we consider the entropy regularized Wasserstein distance, W p,λ ( µ, ν ) , where the search space for the optimal T is instead restricted to a smooth solution close to the extreme points of this linear program.
The regularized problem can then be solved efﬁciently using Sinkhorn iterations (Sinkhorn, 1964), albeit at the cost of some approximation error.
The regularization strength λ ≥controls the accuracy of approximation and recovers the true OT for λ = 0 .
The cost of the Sinkhorn algorithm is only quadratic in n at each iteration.
Recently, Altschuler et al.
(2017) have shown that the Sinkhorn algorithm converges in a number of iterations independent of n , thus resulting in an overall complexity of (cid:101) O
( n 2 /(cid:15) 3 ) for an (cid:15) -accurate solution.
Barycenters.
Further on in our discussion, we will make use of the notion of averaging in the
Wasserstein space.
More precisely, the Wasserstein barycenter, introduced by Agueh & Carlier (2011),
[Figure]
e nd a n g e
r e d s k i n s s p e c
i e s f o ss il s
E l e ph a n t
Mammal is a probability measure that minimizes the sum of ( p -th power)
Wasserstein distances to the given measures.
Formally, given N measures { ν 1 , . . .
, ν N } with corresponding weights η = { η 1 , . .
, η N } ∈ Σ N , the Wasserstein barycenter can be written as B p ( ν 1 , . . .
, ν N ) = arg min µ (cid:80) N i =1 η i W p ( µ, ν i ) p .
We similarly consider the regularized barycenter B p,λ , using entropy regularized Wasserstein distances W p,λ in the above minimization problem, following Cuturi & Doucet (2014).
Employing the method of iterative Bregman projections (Benamou et al., 2015), we obtain an approximation of the solution at a reasonable computational cost.
M ETHODOLOGY
In this section, we deﬁne the distributional estimate that we use to represent each entity.
In view of the guiding example of building text representations, consider each entity to be a word for simplicity.
Distributional Estimate ( P wV ).
For a word w , its distributional estimate is built from a histogram over the set of contexts C , and an embedding of these contexts into a space G .
The histogram essentially measures how likely it is for a word w to occur in a particular context c , i.e., probability p ( w | c ) .
The exact formulation of this distribution is generally intractable and hence it’s common to empirically estimate this by the number of occurrences of the word w in context c , relative to the total frequency of context c in the corpus.
Thus one way to build this histogram is to maintain a co-occurrence matrix between words in our vocabulary and all possible contexts, where each entry indicates how often a word and context occur in an interval (or window) of a ﬁxed size L .
Then, the bin values ( H w ) c ∈C of the histogram (H w ) for a word w , can be viewed as the row corresponding to w in this co-occurrence matrix.
In Section 5, we discuss possible modiﬁcations of the co-occurrence matrix to improve associations and how to reduce the number of bins in the histogram.
The simplest embedding of contexts is into the space of one-hot vectors of all the possible contexts.
However, this induces a lot of sparsity/redundancy in the representation and the distance between such emebddings of contexts does not reﬂect their semantics.
A classical solution would be to instead ﬁnd a dense low-dimensional embedding of contexts that captures the semantics, possibly using techniques such as SVD or deep neural networks.
We denote by V =
( v c ) c ∈C an embedding of the contexts into this low-dimensional space G ⊂ R d , which we refer to as the ground space .
(We will consider example cases of how this metric can be obtained in Sections 6 and 7.)
Combining the histogram H w and the embedding V , we represent the word w by the following empirical distribution:
[Equation]
Recall that δ ( v c ) denotes the Dirac measure at the position v c of the context c .
We refer to this representation (Eq.
2) as
the distributional estimate
of the
word.
Distance.
If we equip the ground space G with a meaningful metric D G , then we can subsequently deﬁne a distance between the representations of two words w i and w j , as the solution to the following
S t a d i u m C o u r t M a t c h F
i r m
Tennis
S t a d i u m C o u r t M a t c h F
i r m S t a d i u m C
o u r t M a t c h F i r m
Football
Law
T e nn i s Foo t b a ll L a w T e nn i s Foo t b a ll L a w T e nn i s Foo t b a ll L a w
T e nn i s Foo t b a ll L a w
Stadium
Match
Firm
Court Stadium
Match
Firm
Court optimal transport problem:
[Equation]
Under this formulation, we call the above distance (Eq. 3) the Context Mover’s Distance (CMD), borrowing the name from Rubner et al.
(2000)’s famous Earth Mover’s Distance in computer vision.
Intuition.
Two words are similar in meaning if the contexts of one word can be easily transported to the contexts of the other, with this cost of transportation being measured by D G .
This idea still remains in line with the distributional hypothesis (Harris, 1954; Rubenstein & Goodenough, 1965) that words in similar contexts have similar meanings, but provides a precise way to quantify it.
Interpretation.
In fact, both elements of the distributional estimate: the histogram and point estimates are closely tied together and required to serve as an effective representation.
For instance, let’s take a toy example and discuss a scenario that might arise when we only have the histogram information.
Consider three words, ‘Tennis’ , ‘Football’ , and ‘Law’ , admitting as contexts { Stadium, Court, Match, Firm } , with respective histograms shown in left part of Figure 2.
Now, if we only took into account the histograms, we would reach the inaccurate conclusion that ‘Tennis’ is closer in semantics to ‘Law’ than to ‘Football’, as there is a considerable overlap at the important context of ‘Court’ for ‘Tennis’ and ‘Law’.
Whereas, together with the point estimate information, it is apparent that the context ‘Stadium’ (in H Football ) can be more cheaply moved to ‘Court’ (in
H Tennis ), but moving ‘Firm’ (in H Law ) to some context in H Tennis is more costly.
Lastly, in the reverse scenario of only considering the point estimates, we would lose much of the uncertainty associated about the contexts in which the words occur.
We illustrate these scenarios in Figure 2.
Roadmap.
In the next section we discuss concretely how this can be applied and for the sake of brevity we restrict to the particular case where contexts consist of single words.
Section 6 details how this framework can be extended to obtain representation for a composition of entities via Wasserstein barycenter.
Lastly in section 7, we utilize the fact that the CMD in 3 is parameterized by ground cost, and show how this ﬂexibility can be used to deﬁne an asymmetric cost measuring entailment.
C ONCRETE F RAMEWORK
Making associations better.
We consider co-occurrences of a word and a context word if the latter appears in a symmetric window of size L around the target word (the word whose distributional estimate we seek).
While each entry of the co-occurrence matrix reﬂects the co-occurrence count of a target word and its context, the counts alone may not necessarily suggest a strong association between the two.
The well-known Positive Pointwise Mutual Information (PPMI) matrix (Church &
Hanks, 1990; Levy et al., 2015) addresses this shortcoming, and is deﬁned as follows:
PPMI ( w, c ) := max(log( p ( w,c ) p ( w ) × p ( c ) ) , 0) .
The PPMI entries are non-zero when the joint probability of co-occurring target and context words is higher than the probability when they are independent.
Typically, these probabilities are estimated from the co-occurrence counts in the corpus.
Further improvements to the PPMI matrix have been suggested, like in Levy & Goldberg (2014b), and following them we make use of a shifted and smoothed PPMI matrix, denoted by SPPMI α,s where α and s denote the smoothing and k-shift parameters 1 .
Overall, these variants of PPMI enable us to extract better semantic associations from the co-occurrence matrix.
Hence, the bin values (at context c ) for the histogram of word w in Eq.
2 can be formulated as: ( H w ) c :=
[Equation]
Computational considerations.
The view of optimal transport between histograms of contexts introduced in Eq.
3 offers a pleasing interpretation (see Figure 1).
However, it might be computationally intractable in its current formulation, since the number of possible contexts can be as large as the size of vocabulary (if the contexts are just single words) or even exponential (if contexts are considered to be phrases, sentences and otherwise).
This is problematic because the Sinkhorn algorithm for regularized optimal transport (Cuturi, 2013, see Section 3) scales roughly quadratically in the histogram size, and the ground cost matrix can also become prohibitive to store in memory.
One possible ﬁx is to instead consider a set of representative contexts in this ground space, for example via clustering.
We believe that with dense low-dimensional embeddings and a meaningful metric between them, we may not require as many contexts as needed before.
For instance, this can be achieved by clustering the contexts with respect to metric D G .
Apart from the computational gain, the clustering will lead to transport between more abstract contexts.
This will although come at the loss of some interpretability.
Now, consider that we have obtained K representative contexts, each covering some part C k of the set of contexts C .
The histogram for word w with respect to these contexts can then be written as ˜ P w ˜ V =
(cid:80)
Kk =1 (˜ H w ) k δ
(˜ v k ) .
Here ˜ v k ∈
˜ V is the point estimate of the k th representative context, and (˜ H w ) k denote the new histogram bin values with respect to the part C k , (˜ H w ) k :=
SPPMI α,s ( w, C k ) (cid:80)
Kk
=
SPPMI α,s ( w, C k ) , with SPPMI α,s ( w,
[Equation]
Summary.
With the above aspects in account and using batched implementations on (Nvidia TitanX)
GPUs, it is possible to compute around 13,700 Wasserstein-distances/second (for histogram of size 100).
Same also holds for barycenters, where we can compute 4,600 barycenters/second for sentences of length 25 and histogram size of 100.
Building this histogram information comes almost for free during the typical learning of embeddings, as in GloVe (Pennington et al., 2014).
A practical take-home message of this work thus is to not throw away the co-occurrence information e.g. when using GloVe, but to instead pass it on to our method.
S ENTENCE R EPRESENTATION WITH C O MB
Traditionally, the goal of this task is to develop a representation for sentences, that captures the semantics conveyed by it.
Most unsupervised representations proposed in the past rely on the composition of vector embeddings for the words, through either additive, multiplicative, or other ways (Mitchell & Lapata, 2008; Arora et al., 2017; Pagliardini et al., 2017).
We propose to represent sentences as probability distributions to better capture the inherent uncertainty and polysemy.
Our belief is that a sentence representation is meaningful if it best captures the simultaneous occurrence of the words in it.
We hypothesize that a sentence, S = ( w 1 , w 2 , . . .
, w N ) , can be efﬁciently represented via the Wasserstein barycenter (see Section 3) of the distributional estimates of its words,
[Equation]
which is itself again a distribution over the ground space G .
We refer to this representation as the
Context Mover’s Barycenters (CoMB) henceforth.
Interestingly, the classical weighted averaging of point-estimates, like Smooth Inverse Frequency (SIF) in (Arora et al., 2017) (without principal component removal), can be seen as a special case of CoMB, when the distribution associated to a word is just a Dirac at its point estimate.
It becomes apparent that having a rich distributional estimate for a word could turn out to be advantageous.
Since with barycenter representation as in Eq. 5, each sentence is also a distribution over contexts, we can utilize the Context Mover’s Distance (CMD) deﬁned in Eq.
3 to deﬁne the distance between two sentences S 1 and S 2 , under a given ground metric D G as follows,
[Equation]
[Table]
Empirical Evaluation.
To evaluate CoMB as an effective sentence representation, we considerdatasets from SemEval semantic textual similarity (STS) tasks (Agirre et al., 2012; 2013; 2014; 2015; 2016).
The objective here is to give a score of how similar two sentences are in their meanings.
As a ground metric ( D G ), we consider the Euclidean distance between the point estimates (embeddings) of words.
We train the GloVe (Pennington et al., 2014) embeddings on the Toronto Book
Corpus (Zhu et al., 2015), and in this process also obtain the histogram information needed for the distributional estimate.
Since GloVe embeddings for similar words are constructed to be close in terms of cosine similarity for similar words, we ﬁnd the representative points by performing K-means clustering with respect to this similarity for K = 300 .
We benchmark 4 our performance against SIF (Smooth Inverse Frequency) from Arora et al.
(2017) who regard it as a “simple but tough-to-beat baseline”, Skip-thought (Kiros et al., 2015), as well as against the plain Bag of Words (BoW) averaging.
Hyperparameters for both SIF and CoMB are tuned on STS16, and the best conﬁgurations so obtained are used for comparison on the other STS tasks.
Table 1 shows that, on all the tasks CoMB outperforms the best variant of SIF on 3 out of 4 tasks in the test set and leads to an overall gain.
Please refer to Tables 5 and 6in Appendix A for detailed results and discussion.
Also, we encourage the reader to check out Section A.6 where we qualitatively analyse CoMB & SIF.
Online computation.
The principal component removal in SIF, which plays a crucial role as evident from the Table 1, has an aspect that typically goes under the rug.
In particular, these principal components are estimated based on embeddings of sentences in the test set.
This way it cleverly utilizes the information shared across sentences in the test set of downstream tasks, but can act as a hindrance in the practical usecase.
For instance, consider the probable scenario (like for a chatbot) where we have to resolve a query about the similarity of two sentences in an online manner.
Here
SIF would undergo a signiﬁcant performance drop, but CoMB is devoid of any issues arising in the online mode while retaining its performance.
Summary and further prospects.
Our focus in these experiments was to compare methods which can build up sentence representations by just obtaining the word vector information.
Hence, we didn’t consider unsupervised methods such as Sent2vec (Pagliardini et al., 2017), that are speciﬁcally trained to work well on sentence similarity.
We observe that CoMB serves as a competitive sentence representation method and remains applicable in the online usecase.
The empirical results are quite encouraging, given the fact that we haven’t even utilized the important property of non-associativity for Wasserstein barycenters (i.e., B p ( µ, B p ( ν, ξ )) (cid:54) =
B p ( B p ( µ, ν ) , ξ ) ).
This implies that we can take into account the word order with various aggregation strategies, like parse trees, and build the sentence representation by recursively computing barycenters phrase by phrase, which although remains beyond the scope of this paper.
Overall, this highlights towards the advantage of having distributional estimates for words, that can be extended to give a meaningful representation of sentences via CoMB in a principled manner.
H YPERNYMY D ETECTION
In linguistics, hypernymy is a relation between words (or sentences) where the semantics of one word (the hyponym ) are contained within that of another word (the hypernym ).
A simple form of this relation is the is-A relation, e.g., cat is an animal .
Hypernymy is a special case of the more general concept of lexical entailment, the detection of which is relevant for tasks such as Question Answering (QA).
Given a database of lexical entailment relations containing, e.g., is-A
(Roger Federer, tennis player) might help a QA system answer “Who is Switzerland’s most successful tennis player?” .
The early unsupervised approaches for this task exploited different linguistic properties of hypernymy (Weeds & Weir, 2003;
Kotlerman et al., 2010; Santus et al., 2014; Rimell, 2014).
While most of these are count-based, point embedding based methods (Chang et al., 2017; Henderson & Popa, 2016) have become more popular in recent years.
Other approaches represent words by Gaussian distributions with KL-divergence as a measure of entailment (Vilnis & McCallum, 2014; Athiwaratkun & Wilson, 2017).
These methods have proven to be powerful, as they not only capture the semantics but also the uncertainty about the contexts in which the word appears.
Therefore, hypernymy detection is a great testbed to verify the effectiveness of our approach (and the particular formulation) to represent each entity by the distribution of its contexts.
To be successful on this task, a method has to consider if all contexts of the hyponym can be encompassed within the contexts of the hypernym.
It can’t just get away by predicting words that are similar.
Hence, it is natural to make use of the Context Mover’s Distance (CMD), Eq. 3, but with an appropriate ground cost that measures entailment relations well.
For this purpose, we utilize a recently proposed method by (Henderson & Popa, 2016; Henderson, 2017) which explicitly models what information is known about a word, by interpreting each entry of the embedding as the degree to which a certain feature is present.
Based on the logical deﬁnition of entailment they derive an operator measuring the entailment similarity between two so-called entailment vectors deﬁned as follows: (cid:126)v i (cid:61) (cid:126)v j = σ ( − (cid:126)v i ) · log σ ( − (cid:126)v j ) , where the sigmoid σ and log are applied component-wise on the embeddings (cid:126)v i ,(cid:126)v j .
Thus, we use as ground cost D Hend.
ij := − (cid:126)v i (cid:61) (cid:126)v j .
This asymmetric ground cost also shows that our framework can be ﬂexibly used with an arbitrary cost function deﬁned on the ground space.
Evaluation.
In total, we evaluated our method on 9 standard datasets: BLESS (Baroni & Lenci, 2011), EVALution (Santus et al., 2015), Benotto (2015), Weeds et al.
(2014), Henderson (2017),
Baroni et al.
(2012), Kotlerman et al.
, Levy et al.
and Turney & Mohammad (2015).
As an evaluation metric, we use average precision AP@all Zhu (2004).
Following Chang et al.
we pushed any OOV (out-of-vocabulary) words in the test data to the bottom of the list, effectively assuming that the word pairs do not have a hypernym relation.
The foremost thing that we would like to check is the beneﬁt of having a distributional estimate in comparison to just the point embeddings.
Here, we observe that by employing CMD along with the entailment embeddings, leads to a signiﬁcant boost on almost all of the datasets, except on Baroni, where the performance is still competitive with the other state of the art methods like Gaussian embeddings.
The more interesting observation is that on some datasets (EVALution, LenciBenotto,
Weeds, Turney) we even outperform or match state-of-the-art performance (cf.
Table 2), by simply using CMD together with this ground cost D Hend.
ij based on the entailment embeddings.
Notably, this approach is not speciﬁc to the entailment vectors from Henderson (2017).
It can possibly be used
[Table]
with any embedding vectors which provide a good measure of the degree of entailment; and a more accurate set of vectors might even further improve the performance
C ONCLUSION
We advocate for representing entities by a distributional estimate on top of any given co-occurrence structure.
For each entity, we jointly consider the histogram information (with its contexts) as well as the point embeddings of the contexts.
We show how this enables the use of optimal transport over distributions of contexts.
Our framework results in an efﬁcient, interpretable and compositional metric to represent and compare entities (e g. words) and groups thereof (e g. sentences), while leveraging existing point embeddings.
We demonstrate its performance on several NLP tasks such as sentence similarity and word entailment detection.
Motivated by the empirical results on the selected tasks, applying the proposed framework on co-occurrence structures beyond NLP is a promising direction.
A S UPPLEMENTARY M ATERIAL A. E XPERIMENTAL D ETAILS
Sentence Representations.
While using the Toronto Book Corpus, we remove the errors caused by crawling and pre-process the corpus by ﬁltering out sentences longer than 300 words, thereby removing a very small portion (500 sentences out of the 70 million sentences).
We utilize the codefrom GloVe for building the vocabulary of size 205513 (obtained by setting min count=10) and the co-occurrence matrix (considering a symmetric window of size 10).
Note that as in GloVe, the contribution from a context word is inversely weighted by the distance to the target word, while computing the co-occurrence.
The vectors obtained via GloVe have 300 dimensions and were trained for 75 iterations at a learning rate of 0.005, other parameters being the default ones.
The performance of these vectors from GloVe was veriﬁed on standard word similarity tasks.
Hypernymy Detection.
The training of the entailment vector is performed on a Wikipedia dump from 2015 with 1.7B tokens that have been tokenized using the Stanford NLP library (Manning et al., 2014).
In our experiments, we use a vocabulary with a size of 80’000 and word embeddings with 200 dimensions and 100 cluster centers.
We followed the same training procedure as described in
Henderson (2017) and were able to reproduce their scores on the hypernymy detection task.
A. PPMI D ETAILS
Formulation and Variants.
Typically, the probabilities used in PMIare estimated from the cooccurrence counts #( w, c ) in the corpus and lead to
PPMI (
[Equation]
, c
[Equation]
=
[Equation]
[Equation] [Equation]
(cid:19)
[Equation]
(cid:19)
[Equation]
(7) where, #( w ) =
(cid:80) c
#( w, c ) , #( c ) =
(cid:80) w #( w, c ) and | Z | = (cid:80) w (cid:80) c
#( w, c ) .
Also, it is known that PPMI is biased towards infrequent words and assigns them a higher value.
A common solution is to smoothen 8 the context probabilities by raising them to an exponent of α lying between 0 and 1 .
Levy & Goldberg (2014b) have also suggested the use of the shifted PPMI (SPPMI) matrix where the shift 9 by log( s ) acts like a prior on the probability of co-occurrence of target and context pairs.
These variants of PPMI enable us to extract better semantic associations from the co-occurrence matrix.
Finally, we have
[Equation]
Computational aspect.
We utilize the sparse matrix support of Scipy 10 for efﬁciently carrying out all the PPMI computations.
PPMI Column Normalizations.
In certain cases, when the PPMI contributions towards the partitions (or clusters) have a large variance, it can be helpful to consider the fraction of C k ’s SPPMI (Eq. 8, 9) that has been used towards a word w , instead of aggregate values used in 4.
Otherwise the process of making the histogram unit sum might misrepresent the actual underlying contribution.
We call this PPMI column normalization ( β ).
In other words, the intuition is that the normlization will balance the effect of a possible non-uniform spread in total PPMI across the clusters.
We observe that setting β to 0 .
5 or 1 help in boosting performance on the STS tasks.
The basic form of column normalization is shown in 9.
[Equation]
with
[Equation]
Another possibility while considering the normalization to have an associated parameter β that can interpolate between the above normalization and normalization with respcect to cluster size.
[Equation]
where
[Equation]
In particular, when β = 1 , we recover the equation for histograms as in 9, and β = 0 would imply normalization with respect to cluster sizes.
A. O PTIMAL T RANSPORT
Implementation aspects.
We make use of the Python Optimal Transport (POT) 11 for performing the computation of Wasserstein distances and barycenters on CPU.
For more efﬁcient GPU implementation, we built custom implementation using PyTorch.
We also implement a batched version for barycenter computation, which to the best of our knowledge has not been done in the past.
The batched barycenter computation relies on a viewing computations in the form of block-diagonal matrices.
As an example, this batched mode can compute around 200 barycenters in 0.09 seconds, where each barycenter is of 50 histograms (of size 100) and usually gives a speedup of about 10x.
Scalability.
For further scalability, an alternative is to consider stochastic optimal transport techniques (Genevay et al., 2016).
Here, the idea would be to randomly sample a subset of contexts from the distributional estimate while considering this transport.
Stability of Sinkhorn Iterations.
For all our computations involving optimal transport, we typically use λ around 0 .
1 and make use of log or median normalization as common in POT to stabilize the Sinkhorn iterations.
Also, we observe that clipping the ground metric matrix (if it exceeds a particular large threshold) also sometimes results in performance gains.
Value of p .
It has been shown in Agueh & Carlier (2011) that when the underlying space is
Euclidean and p = 2 , there exists a unique minimizer to the Wasserstein barycenter problem.
But, since we are anyways solving the regularized Wasserstein barycenter (Cuturi & Doucet, 2014) problem over here instead of the exact one, the particular value of p seems less of an issue.
Empirically in the sentence similarity experiments, we have observed p = 1 to perform better than p = 2 (by about 2-3 points).
A. C LUSTERING .
For clustering, we make use of kmcuda’s 12 efﬁcient implementation of K-Means algorithm on GPUs.
A. S OFTWARE R ELEASE
Core code and histograms.
We plan to make all our code (for all these parts) and our pre-computed histograms (for the mentioned datasets) publicly available on GitHub soon.
Standard evaluation suite for Hypernymy.
To ease the evaluation pipeline, we have collected the most common benchmark datasets and compiled the code for assessing a model’s performance on hypernymy detection or directionality into a Python package, called HypEval , which is publicly available.
This also handles OOV (out-of-vocabulary) pairs in a standardized manner and allows for efﬁcient, batched evaluation on GPU.
A. Q UALITATIVE A NALYSIS OF S ENTENCE R EPRESENTATIONS
Here, we would like to qualitatively probe the kind of results obtained when computing Wasserstein barycenter of the distributional estimates, in particular, when using CoMB to represent sentences.
To this end
[Table]
we consider a few simple sentences and ﬁnd the closest word in the vocabulary for CoMB (with respect to CMD) and contrast it to SIF with cosine distance
[Table]
Observations.
We ﬁnd that closest neighbors (see Table3 ) for CoMB consist of relatively more diverse set of words which ﬁt well in the context of given sentence.
For example, take the sentence “i love her”, where CoMB captures a wide range of contexts, for example, “i actually love her”, “i love her because ”, “i doubt her love” and more.
Also for an ambiguous sentence “he lives in europe for”, the obtained closest neighbors for CoMB include: ‘decades’, ‘masters’, ‘majority’, ‘commerce’ , etc., while with SIF the closest neighbors are mostly words similar to one of the query words.
Further, if you look at the last three sentences in the Table3, the ﬁrst closest neighbor for CoMB even acts as a good next word for the given query.
This suggests that CoMB might perform well on the task of sentence completion, but this additional evaluation is beyond the scope of this paper.
A. D ETAILED R ESULTS
Detailed results of the hypernymy detection and sentence representation experiments are listed on the following pages.
A.7. H YPERNYMY D ETECTION
[Table]
Henderson
[Table] [Table]
CMD ( α =0 .
[Table]
s =1 ) + D Hend.
[Table]
CMD ( α =0 .
15 , s =5 ) + D Hend.
[Table]
CMD ( α =0 .
15 , s =15 )
[Table] [Table]
CMD ( α =0 .
[Table]
s =1 )
+
[Table] [Table]
CMD ( α =
[Table] [Table]
[Table]
CMD ( α =
[Table]
=15 )
[Table] [Table]
CMD ( α =0 .
15 , s =1 ) + D Hend.
[Table]
CMD ( α =0 .
15 , s =5 ) + D Hend.
[Table]
A.7. S ENTENCE R EPRESENTATION
We observe empirically that the PPMI smoothing parameter α , which balances the bias of PPMI towards rare words, plays an important role.
While its ideal value would vary on each task
[Table]
we found the settings mentioned in the Table 5 to work well uniformly across the above spectrum of tasks
[Table] [Table]
[Table]
A. E FFECT OF NUMBER OF CLUSTERS
Here, we analyse the impact of number of clusters on the performance of Context Mover’s Barycenters (CoMB) for the sentence similarity experiments (c.f. Section 6).
In particular, we look at the three best performing variants (A, B, C) on the validation set (STS 16) as well as averaged across them.
We observe in Figure 3 that on average the performance signiﬁcantly improves when the number of clusters are increased until around K = 300 , and beyond that mostly plateaus ( ± 0.5).
But, as can be seen for variants B and C the performance typically continues to rise until K = 500 .
It seems that the amount of PPMI column normalization ( β = 0 .
5 vs β = 1 ) might be at play here.
Overall, going from K = 300 to K = 500 comes at the cost of increased computation time, and doesn’t lead to a substantial gain in performance.
This reﬂects the tradeoff between accuracy and efﬁciency.
Hence, we stick to K =
300 for our results on sentence similarity tasks.
Such a trend seems to be inline with the ideal case where we wouldn’t need to do any clustering and just take all possible contexts into account.
Thus, it suggests that better ways (other than clustering) to deal with this problem might further boost the performance.