C ONTEXT M OVER ’ S D ISTANCE & B ARYCENTERS : O PTIMAL TRANSPORT OF CONTEXTS FOR BUILDING REPRESENTATIONS
Anonymous authors
Paper under double-blind review
A BSTRACT
We propose a uniﬁed framework for building unsupervised representations of entities and their compositions, by viewing each entity as a histogram (or distribution) over its contexts.
This enables us to take advantage of optimal transport and construct representations that effectively harness the geometry of the underlying space containing the contexts.
Our method captures uncertainty via modelling the entities as distributions and simultaneously provides interpretability with the optimal transport map, hence giving a novel perspective for building rich and powerful feature representations.
As a guiding example, we formulate unsupervised representations for text, and demonstrate it on tasks such as sentence similarity and word entailment detection.
Empirical results show strong advantages gained through the proposed framework.
This approach can potentially be used for any unsupervised or supervised problem (on text or other modalities) with a co-occurrence structure, such as any sequence data.
The key tools at the core of this framework are Wasserstein distances and Wasserstein barycenters, hence raising the question from our title.
I NTRODUCTION
One of the main driving factors behind the recent successes in machine learning and natural language processing has been the development of better representation methods for data modalities.
Examples include continuous vector representations for language (Mikolov et al., 2013; Pennington et al., 2014),
Convolutional Neural Network (CNN) based text representations (Collobert & Weston, 2008; Kim, 2014; Kalchbrenner et al., 2014; Severyn & Moschitti, 2015; Deriu et al., 2017), or via other neural architectures such as RNNs, LSTMs (Hochreiter & Schmidhuber, 1997; Kiros et al., 2015), all sharing one central idea – to map input entities to dense vector embeddings lying in a low-dimensional latent space where the semantics of the inputs are preserved.
While these existing methods represent each entity of interest (e.g., a word) directly as a single point in space (i.e., its embedding vector), we here propose a fundamentally different approach.
We focus on co-occurrence information of the entities and their contexts (e.g. context words), and leverage embeddings of the contexts instead of the original entities.
So instead of a single point per entity, we represent each entity by the histogram of its contexts , where the contexts themselves are represented as points in a suitable metric space.
This allows us to cast the distance between histograms associated with the contexts as an instance of the optimal transport problem (Monge, 1781; Kantorovich, 1942;
Villani, 2008).
Our resulting framework then intuitively seeks to minimize the cost of moving the contexts of a given entity to the contexts of another, which motivates the naming Context Mover’s Distance (CMD).
Note that the contexts here can be words, phrases, sentences, or any generic entities co-occurring with our entities to be represented, and these entities further could be of various kinds, including e.g., products such as movies or web-advertisements (Grbovic et al., 2015), nodes in a graph (Grover & Leskovec, 2016), sequence data, or any other entities (Wu et al., 2017).
Any co-occurrence structure will allow construction of the histogram information, which is the crucial building block of our approach.
The main motivation for our proposed approach here comes from the domain of natural language, where the entities (words, phrases, or sentences) generally have different semantics depending on the context under which they are present.
Hence, it is important that we consider representations that are able to effectively capture such inherent uncertainty and polysemy, and we will argue that histograms (or probability distributions) over embeddings allows to capture more of this information compared to point-wise embeddings alone.
We will call this histogram over contexts embeddings as the distributional estimate of our entity of interest, while we refer to the individual embeddings of contexts as point estimates .
The connection to optimal transport at the level of entities and contexts paves the way to make better use of its vast toolkit (like Wasserstein distances, barycenters, barycentric coordinates, etc.) for applications in NLP, which in the past has primarily been restricted to document distances of original words (Kusner et al., 2015; Huang et al., 2016), as opposed to contexts.
Thanks to the entropic regularization introduced by Cuturi (2013), optimal transport computations can be carried out efﬁciently in a parallel and batched manner on GPUs.
Contributions:
• Employing the notion of optimal transport of contexts as a distance measure, we illustrate how our framework can be of beneﬁt for various important tasks, including word and sentence representations, sentence similarity, as well as hypernymy (entailment) detection.
The method is static and does not require any additional learning, and can be readily used on top of existing embedding methods.
• The resulting representations via the transport map give a clear interpretation of the resulting distance (see also Figure 1), on top of the co-occurrence information.
• Our Context Mover’s Distance can be used to measure any kind of distance (even asymmetric) between words, by deﬁning a suitable underlying cost on the movement of contexts, which we show can lead to a state-of-the-art metric for word entailment.
• Deﬁning the transport over contexts has the added beneﬁt that the representations are compositional - they directly extend from entities to groups of entities (of any size), such as from word to sentence representations.
To this end, we utilize the notion of Wasserstein barycenters, which to the best of our knowledge has never been considered in the past.
The proposed framework is not speciﬁc to words or sentences but can allow building unsupervised representations for any entity and composition of entities, where a co-occurrence structure can be devised between entities and their contexts, and we leave this direction for a future work.
R ELATED W ORK
Most of the previous work in building representations for natural language has been focused towards vector space models, in particular, popularized through the groundbreaking work in
Word2vec (Mikolov et al., 2013) and GloVe (Pennington et al., 2014).
The key idea in these models has been to map words which are similar in meaning to nearby points in a latent space.
Based on which, many works (Levy & Goldberg, 2014a; Melamud et al., 2015; Bojanowski et al., 2016) have suggested specializing the embeddings to capture some particular information required for the task at hand.
One of the problems that still persists is the inability to capture, within just a point embedding, the various semantics and uncertainties associated with the occurrence of a particular word (Huang et al., 2012; Guo et al., 2014).
A recent line of work has proposed the view to represent words with Gaussian distributions or mixtures of Gaussian distributions (Vilnis & McCallum, 2014; Athiwaratkun & Wilson, 2017), or hyperbolic cones (Ganea et al., 2018) for this purpose.
Also, concurrent works by Muzellec & Cuturi (2018) and Sun et al.
(2018) have suggested using elliptical and Gaussian distributions endowed with a Wasserstein metric respectively.
While these already provide richer information than typical vector embeddings, their form restricts what could be gained by allowing for arbitrary distributions.
In addition, hyperbolic embeddings (Nickel & Kiela, 2017; Ganea et al., 2018) are so far restricted to supervised tasks (and even elliptical embeddings (Muzellec & Cuturi, 2018) to the most extent), not allowing unsupervised representation learning as in the focus of the paper here.
To this end, we propose to represent each word with a distributional estimate (i.e., histogram over context embeddings), that inherently relies upon the empirically obtained co-occurrence information of a word and its contexts.
Hence, this naturally allows for the use of optimal transport (or Wasserstein metric) in the space containing the contexts, and leads to an interpretation (Figure 1) which is not available in the above approaches.
Amongst the few explorations of optimal transport in NLP, i.e., document distances (Kusner et al., 2015; Huang et al., 2016), topic modelling (Rolet et al., 2016; Xu et al., 2018), document clustering (Ye et al., 2017), bilingual lexicon induction (Zhang et al., 2017), or orthogonal Procrustes mapping (Grave et al., 2018), the focus has been on transporting words directly.
For example, the
Word Mover’s Distance (Kusner et al., 2015) casts ﬁnding the distance between documents as an optimal transport problem between their bag of words representation.
Our approach is different as we consider the transport over contexts instead, and use it to propose a representation for words or entities.
This provides the added ﬂexibility to establish any kind of distance between entities and extend the representation to composition of entities in a principled manner, as will be illustrated further through the examples of entailment detection and sentence representation respectively.
B ACKGROUND ON O PTIMAL T RANSPORT
Optimal Transport (OT) provides a way to compare two probability distributions deﬁned over a space G (commonly known as the ground space), given an underlying distance or more generally a cost of moving one point to another in the ground space.
In other terms, it lifts a distance between points to a distance between distributions.
Other methods of comparing distributions, such as Kullback-Liebler (KL), squared Hellinger, etc., only focus on the probability mass values, thus ignoring the geometry of the ground space: something which we utilize throughout this work via OT.
Also, some divergences like KL are not deﬁned when the supports of distributions under comparison don’t match.
Hence, we give a short yet formal background on OT in the discrete case.
Linear Program Formulation.
Consider an empirical probability measure of the form µ = (cid:80) ni
=1 a i δ ( x i ) where
X =
( x 1 , . .
, x n ) ∈ G n , δ ( x ) denotes the Dirac (unit mass) distribution at point x ∈ G , and ( a 1 , . .
. , a n ) lives in the probability simplex Σ n := (cid:8) p ∈ R n + | (cid:80)
[Equation]
1 p i
[Equation] [Equation]
Now given a second empirical measure, ν = (cid:80) mj
=1 b j δ ( y j ) , with Y =
( y 1 , . . .
, y m ) ∈ G m , and ( b 1 , . . .
, b m ) ∈ Σ m , and if the ground cost of moving from point x i to y j is denoted by M ij , then the
Optimal Transport distance between µ and ν is the solution to the following linear program.
[Equation]
Here, the optimal T ∈ R n × m is referred to as the transportation matrix : T ij denotes the optimal amount of mass to move from point x i to point y i .
Intuitively, OT is concerned with the problem of moving goods from factories to shops in such a way that all the demands are satisﬁed and the overall transportation cost is minimal.
Distance.
When G = R d and the cost is deﬁned with respect to a metric D G over G (i.e., M ij =
D G ( x i , y j ) p for any i, j ), OT deﬁnes a distance between empirical probability distributions.
This is the p -Wasserstein distance, deﬁned as W p ( µ, ν ) :
=
OT ( µ, ν ; D p G ) 1 /p .
In most cases, we are only concerned with the case where p = 1 or 2 .
The cost of exactly solving OT problem scales at least in O ( n 3 log( n ))
( n being the cardinality of the support of the empirical measure) when using network simplex or interior point methods.
Following
Cuturi (2013), we consider the entropy regularized Wasserstein distance, W p,λ ( µ, ν ) , where the search space for the optimal T is instead restricted to a smooth solution close to the extreme points of this linear program.
The regularized problem can then be solved efﬁciently using Sinkhorn iterations (Sinkhorn, 1964), albeit at the cost of some approximation error.
The regularization strength λ ≥controls the accuracy of approximation and recovers the true OT for λ = 0 .
The cost of the Sinkhorn algorithm is only quadratic in n at each iteration.
Recently, Altschuler et al.
(2017) have shown that the Sinkhorn algorithm converges in a number of iterations independent of n , thus resulting in an overall complexity of (cid:101) O
( n 2 /(cid:15) 3 ) for an (cid:15) -accurate solution.
Barycenters.
Further on in our discussion, we will make use of the notion of averaging in the
Wasserstein space.
More precisely, the Wasserstein barycenter, introduced by Agueh & Carlier (2011),
[Figure]
e nd a n g e
r e d s k i n s s p e c
i e s f o ss il s
E l e ph a n t
Mammal is a probability measure that minimizes the sum of ( p -th power)
Wasserstein distances to the given measures.
Formally, given N measures { ν 1 , . . .
, ν N } with corresponding weights η = { η 1 , . .
, η N } ∈ Σ N , the Wasserstein barycenter can be written as B p ( ν 1 , . . .
, ν N ) = arg min µ (cid:80) N i =1 η i W p ( µ, ν i ) p .
We similarly consider the regularized barycenter B p,λ , using entropy regularized Wasserstein distances W p,λ in the above minimization problem, following Cuturi & Doucet (2014).
Employing the method of iterative Bregman projections (Benamou et al., 2015), we obtain an approximation of the solution at a reasonable computational cost.
M ETHODOLOGY
In this section, we deﬁne the distributional estimate that we use to represent each entity.
In view of the guiding example of building text representations, consider each entity to be a word for simplicity.
Distributional Estimate ( P wV ).
For a word w , its distributional estimate is built from a histogram over the set of contexts C , and an embedding of these contexts into a space G .
The histogram essentially measures how likely it is for a word w to occur in a particular context c , i.e., probability p ( w | c ) .
The exact formulation of this distribution is generally intractable and hence it’s common to empirically estimate this by the number of occurrences of the word w in context c , relative to the total frequency of context c in the corpus.
Thus one way to build this histogram is to maintain a co-occurrence matrix between words in our vocabulary and all possible contexts, where each entry indicates how often a word and context occur in an interval (or window) of a ﬁxed size L .
Then, the bin values ( H w ) c ∈C of the histogram (H w ) for a word w , can be viewed as the row corresponding to w in this co-occurrence matrix.
In Section 5, we discuss possible modiﬁcations of the co-occurrence matrix to improve associations and how to reduce the number of bins in the histogram.
The simplest embedding of contexts is into the space of one-hot vectors of all the possible contexts.
However, this induces a lot of sparsity/redundancy in the representation and the distance between such emebddings of contexts does not reﬂect their semantics.
A classical solution would be to instead ﬁnd a dense low-dimensional embedding of contexts that captures the semantics, possibly using techniques such as SVD or deep neural networks.
We denote by V =
( v c ) c ∈C an embedding of the contexts into this low-dimensional space G ⊂ R d , which we refer to as the ground space .
(We will consider example cases of how this metric can be obtained in Sections 6 and 7.)
Combining the histogram H w and the embedding V , we represent the word w by the following empirical distribution:
[Equation]
Recall that δ ( v c ) denotes the Dirac measure at the position v c of the context c .
We refer to this representation (Eq.
2) as
the distributional estimate
of the
word.
Distance.
If we equip the ground space G with a meaningful metric D G , then we can subsequently deﬁne a distance between the representations of two words w i and w j , as the solution to the following
S t a d i u m C o u r t M a t c h F
i r m
Tennis
S t a d i u m C o u r t M a t c h F
i r m S t a d i u m C
o u r t M a t c h F i r m
Football
Law
T e nn i s Foo t b a ll L a w T e nn i s Foo t b a ll L a w T e nn i s Foo t b a ll L a w
T e nn i s Foo t b a ll L a w
Stadium
Match
Firm
Court Stadium
Match
Firm
Court optimal transport problem:
[Equation]
Under this formulation, we call the above distance (Eq. 3) the Context Mover’s Distance (CMD), borrowing the name from Rubner et al.
(2000)’s famous Earth Mover’s Distance in computer vision.
Intuition.
Two words are similar in meaning if the contexts of one word can be easily transported to the contexts of the other, with this cost of transportation being measured by D G .
This idea still remains in line with the distributional hypothesis (Harris, 1954; Rubenstein & Goodenough, 1965) that words in similar contexts have similar meanings, but provides a precise way to quantify it.
Interpretation.
In fact, both elements of the distributional estimate: the histogram and point estimates are closely tied together and required to serve as an effective representation.
For instance, let’s take a toy example and discuss a scenario that might arise when we only have the histogram information.
Consider three words, ‘Tennis’ , ‘Football’ , and ‘Law’ , admitting as contexts { Stadium, Court, Match, Firm } , with respective histograms shown in left part of Figure 2.
Now, if we only took into account the histograms, we would reach the inaccurate conclusion that ‘Tennis’ is closer in semantics to ‘Law’ than to ‘Football’, as there is a considerable overlap at the important context of ‘Court’ for ‘Tennis’ and ‘Law’.
Whereas, together with the point estimate information, it is apparent that the context ‘Stadium’ (in H Football ) can be more cheaply moved to ‘Court’ (in
H Tennis ), but moving ‘Firm’ (in H Law ) to some context in H Tennis is more costly.
Lastly, in the reverse scenario of only considering the point estimates, we would lose much of the uncertainty associated about the contexts in which the words occur.
We illustrate these scenarios in Figure 2.
Roadmap.
In the next section we discuss concretely how this can be applied and for the sake of brevity we restrict to the particular case where contexts consist of single words.
Section 6 details how this framework can be extended to obtain representation for a composition of entities via Wasserstein barycenter.
Lastly in section 7, we utilize the fact that the CMD in 3 is parameterized by ground cost, and show how this ﬂexibility can be used to deﬁne an asymmetric cost measuring entailment.
C ONCRETE F RAMEWORK
Making associations better.
We consider co-occurrences of a word and a context word if the latter appears in a symmetric window of size L around the target word (the word whose distributional estimate we seek).
While each entry of the co-occurrence matrix reﬂects the co-occurrence count of a target word and its context, the counts alone may not necessarily suggest a strong association between the two.
The well-known Positive Pointwise Mutual Information (PPMI) matrix (Church &
Hanks, 1990; Levy et al., 2015) addresses this shortcoming, and is deﬁned as follows:
PPMI ( w, c ) := max(log( p ( w,c ) p ( w ) × p ( c ) ) , 0) .
The PPMI entries are non-zero when the joint probability of co-occurring target and context words is higher than the probability when they are independent.
Typically, these probabilities are estimated from the co-occurrence counts in the corpus.
Further improvements to the PPMI matrix have been suggested, like in Levy & Goldberg (2014b), and following them we make use of a shifted and smoothed PPMI matrix, denoted by SPPMI α,s where α and s denote the smoothing and k-shift parameters 1 .
Overall, these variants of PPMI enable us to extract better semantic associations from the co-occurrence matrix.
Hence, the bin values (at context c ) for the histogram of word w in Eq.
2 can be formulated as: ( H w ) c :=
[Equation]
Computational considerations.
The view of optimal transport between histograms of contexts introduced in Eq.
3 offers a pleasing interpretation (see Figure 1).
However, it might be computationally intractable in its current formulation, since the number of possible contexts can be as large as the size of vocabulary (if the contexts are just single words) or even exponential (if contexts are considered to be phrases, sentences and otherwise).
This is problematic because the Sinkhorn algorithm for regularized optimal transport (Cuturi, 2013, see Section 3) scales roughly quadratically in the histogram size, and the ground cost matrix can also become prohibitive to store in memory.
One possible ﬁx is to instead consider a set of representative contexts in this ground space, for example via clustering.
We believe that with dense low-dimensional embeddings and a meaningful metric between them, we may not require as many contexts as needed before.
For instance, this can be achieved by clustering the contexts with respect to metric D G .
Apart from the computational gain, the clustering will lead to transport between more abstract contexts.
This will although come at the loss of some interpretability.
Now, consider that we have obtained K representative contexts, each covering some part C k of the set of contexts C .
The histogram for word w with respect to these contexts can then be written as ˜ P w ˜ V =
(cid:80)
Kk =1 (˜ H w ) k δ
(˜ v k ) .
Here ˜ v k ∈
˜ V is the point estimate of the k th representative context, and (˜ H w ) k denote the new histogram bin values with respect to the part C k , (˜ H w ) k :=
SPPMI α,s ( w, C k ) (cid:80)
Kk
=
SPPMI α,s ( w, C k ) , with SPPMI α,s ( w,
[Equation]
Summary.
With the above aspects in account and using batched implementations on (Nvidia TitanX)
GPUs, it is possible to compute around 13,700 Wasserstein-distances/second (for histogram of size 100).
Same also holds for barycenters, where we can compute 4,600 barycenters/second for sentences of length 25 and histogram size of 100.
Building this histogram information comes almost for free during the typical learning of embeddings, as in GloVe (Pennington et al., 2014).
A practical take-home message of this work thus is to not throw away the co-occurrence information e.g. when using GloVe, but to instead pass it on to our method.
S ENTENCE R EPRESENTATION WITH C O MB
Traditionally, the goal of this task is to develop a representation for sentences, that captures the semantics conveyed by it.
Most unsupervised representations proposed in the past rely on the composition of vector embeddings for the words, through either additive, multiplicative, or other ways (Mitchell & Lapata, 2008; Arora et al., 2017; Pagliardini et al., 2017).
We propose to represent sentences as probability distributions to better capture the inherent uncertainty and polysemy.
Our belief is that a sentence representation is meaningful if it best captures the simultaneous occurrence of the words in it.
We hypothesize that a sentence, S = ( w 1 , w 2 , . . .
, w N ) , can be efﬁciently represented via the Wasserstein barycenter (see Section 3) of the distributional estimates of its words,
[Equation]
which is itself again a distribution over the ground space G .
We refer to this representation as the
Context Mover’s Barycenters (CoMB) henceforth.
Interestingly, the classical weighted averaging of point-estimates, like Smooth Inverse Frequency (SIF) in (Arora et al., 2017) (without principal component removal), can be seen as a special case of CoMB, when the distribution associated to a word is just a Dirac at its point estimate.
It becomes apparent that having a rich distributional estimate for a word could turn out to be advantageous.
Since with barycenter representation as in Eq. 5, each sentence is also a distribution over contexts, we can utilize the Context Mover’s Distance (CMD) deﬁned in Eq.
3 to deﬁne the distance between two sentences S 1 and S 2 , under a given ground metric D G as follows,
[Equation]
[Table]
Empirical Evaluation.
To evaluate CoMB as an effective sentence representation, we considerdatasets from SemEval semantic textual similarity (STS) tasks (Agirre et al., 2012; 2013; 2014; 2015; 2016).
The objective here is to give a score of how similar two sentences are in their meanings.
As a ground metric ( D G ), we consider the Euclidean distance between the point estimates (embeddings) of words.
We train the GloVe (Pennington et al., 2014) embeddings on the Toronto Book
Corpus (Zhu et al., 2015), and in this process also obtain the histogram information needed for the distributional estimate.
Since GloVe embeddings for similar words are constructed to be close in terms of cosine similarity for similar words, we ﬁnd the representative points by performing K-means clustering with respect to this similarity for K = 300 .
We benchmark 4 our performance against SIF (Smooth Inverse Frequency) from Arora et al.
(2017) who regard it as a “simple but tough-to-beat baseline”, Skip-thought (Kiros et al., 2015), as well as against the plain Bag of Words (BoW) averaging.
Hyperparameters for both SIF and CoMB are tuned on STS16, and the best conﬁgurations so obtained are used for comparison on the other STS tasks.
Table 1 shows that, on all the tasks CoMB outperforms the best variant of SIF on 3 out of 4 tasks in the test set and leads to an overall gain.
Please refer to Tables 5 and 6in Appendix A for detailed results and discussion.
Also, we encourage the reader to check out Section A.6 where we qualitatively analyse CoMB & SIF.
Online computation.
The principal component removal in SIF, which plays a crucial role as evident from the Table 1, has an aspect that typically goes under the rug.
In particular, these principal components are estimated based on embeddings of sentences in the test set.
This way it cleverly utilizes the information shared across sentences in the test set of downstream tasks, but can act as a hindrance in the practical usecase.
For instance, consider the probable scenario (like for a chatbot) where we have to resolve a query about the similarity of two sentences in an online manner.
Here
SIF would undergo a signiﬁcant performance drop, but CoMB is devoid of any issues arising in the online mode while retaining its performance.
Summary and further prospects.
Our focus in these experiments was to compare methods which can build up sentence representations by just obtaining the word vector information.
Hence, we didn’t consider unsupervised methods such as Sent2vec (Pagliardini et al., 2017), that are speciﬁcally trained to work well on sentence similarity.
We observe that CoMB serves as a competitive sentence representation method and remains applicable in the online usecase.
The empirical results are quite encouraging, given the fact that we haven’t even utilized the important property of non-associativity for Wasserstein barycenters (i.e., B p ( µ, B p ( ν, ξ )) (cid:54) =
B p ( B p ( µ, ν ) , ξ ) ).
This implies that we can take into account the word order with various aggregation strategies, like parse trees, and build the sentence representation by recursively computing barycenters phrase by phrase, which although remains beyond the scope of this paper.
Overall, this highlights towards the advantage of having distributional estimates for words, that can be extended to give a meaningful representation of sentences via CoMB in a principled manner.
H YPERNYMY D ETECTION
In linguistics, hypernymy is a relation between words (or sentences) where the semantics of one word (the hyponym ) are contained within that of another word (the hypernym ).
A simple form of this relation is the is-A relation, e.g., cat is an animal .
Hypernymy is a special case of the more general concept of lexical entailment, the detection of which is relevant for tasks such as Question Answering (QA).
Given a database of lexical entailment relations containing, e.g., is-A
(Roger Federer, tennis player) might help a QA system answer “Who is Switzerland’s most successful tennis player?” .
The early unsupervised approaches for this task exploited different linguistic properties of hypernymy (Weeds & Weir, 2003;
Kotlerman et al., 2010; Santus et al., 2014; Rimell, 2014).
While most of these are count-based, point embedding based methods (Chang et al., 2017; Henderson & Popa, 2016) have become more popular in recent years.
Other approaches represent words by Gaussian distributions with KL-divergence as a measure of entailment (Vilnis & McCallum, 2014; Athiwaratkun & Wilson, 2017).
These methods have proven to be powerful, as they not only capture the semantics but also the uncertainty about the contexts in which the word appears.
Therefore, hypernymy detection is a great testbed to verify the effectiveness of our approach (and the particular formulation) to represent each entity by the distribution of its contexts.
To be successful on this task, a method has to consider if all contexts of the hyponym can be encompassed within the contexts of the hypernym.
It can’t just get away by predicting words that are similar.
Hence, it is natural to make use of the Context Mover’s Distance (CMD), Eq. 3, but with an appropriate ground cost that measures entailment relations well.
For this purpose, we utilize a recently proposed method by (Henderson & Popa, 2016; Henderson, 2017) which explicitly models what information is known about a word, by interpreting each entry of the embedding as the degree to which a certain feature is present.
Based on the logical deﬁnition of entailment they derive an operator measuring the entailment similarity between two so-called entailment vectors deﬁned as follows: (cid:126)v i (cid:61) (cid:126)v j = σ ( − (cid:126)v i ) · log σ ( − (cid:126)v j ) , where the sigmoid σ and log are applied component-wise on the embeddings (cid:126)v i ,(cid:126)v j .
Thus, we use as ground cost D Hend.
ij := − (cid:126)v i (cid:61) (cid:126)v j .
This asymmetric ground cost also shows that our framework can be ﬂexibly used with an arbitrary cost function deﬁned on the ground space.
Evaluation.
In total, we evaluated our method on 9 standard datasets: BLESS (Baroni & Lenci, 2011), EVALution (Santus et al., 2015), Benotto (2015), Weeds et al.
(2014), Henderson (2017),
Baroni et al.
(2012), Kotlerman et al.
, Levy et al.
and Turney & Mohammad (2015).
As an evaluation metric, we use average precision AP@all Zhu (2004).
Following Chang et al.
we pushed any OOV (out-of-vocabulary) words in the test data to the bottom of the list, effectively assuming that the word pairs do not have a hypernym relation.
The foremost thing that we would like to check is the beneﬁt of having a distributional estimate in comparison to just the point embeddings.
Here, we observe that by employing CMD along with the entailment embeddings, leads to a signiﬁcant boost on almost all of the datasets, except on Baroni, where the performance is still competitive with the other state of the art methods like Gaussian embeddings.
The more interesting observation is that on some datasets (EVALution, LenciBenotto,
Weeds, Turney) we even outperform or match state-of-the-art performance (cf.
Table 2), by simply using CMD together with this ground cost D Hend.
ij based on the entailment embeddings.
Notably, this approach is not speciﬁc to the entailment vectors from Henderson (2017).
It can possibly be used
[Table]
with any embedding vectors which provide a good measure of the degree of entailment; and a more accurate set of vectors might even further improve the performance
C ONCLUSION
We advocate for representing entities by a distributional estimate on top of any given co-occurrence structure.
For each entity, we jointly consider the histogram information (with its contexts) as well as the point embeddings of the contexts.
We show how this enables the use of optimal transport over distributions of contexts.
Our framework results in an efﬁcient, interpretable and compositional metric to represent and compare entities (e g. words) and groups thereof (e g. sentences), while leveraging existing point embeddings.
We demonstrate its performance on several NLP tasks such as sentence similarity and word entailment detection.
Motivated by the empirical results on the selected tasks, applying the proposed framework on co-occurrence structures beyond NLP is a promising direction.
A S UPPLEMENTARY M ATERIAL A. E XPERIMENTAL D ETAILS
Sentence Representations.
While using the Toronto Book Corpus, we remove the errors caused by crawling and pre-process the corpus by ﬁltering out sentences longer than 300 words, thereby removing a very small portion (500 sentences out of the 70 million sentences).
We utilize the codefrom GloVe for building the vocabulary of size 205513 (obtained by setting min count=10) and the co-occurrence matrix (considering a symmetric window of size 10).
Note that as in GloVe, the contribution from a context word is inversely weighted by the distance to the target word, while computing the co-occurrence.
The vectors obtained via GloVe have 300 dimensions and were trained for 75 iterations at a learning rate of 0.005, other parameters being the default ones.
The performance of these vectors from GloVe was veriﬁed on standard word similarity tasks.
Hypernymy Detection.
The training of the entailment vector is performed on a Wikipedia dump from 2015 with 1.7B tokens that have been tokenized using the Stanford NLP library (Manning et al., 2014).
In our experiments, we use a vocabulary with a size of 80’000 and word embeddings with 200 dimensions and 100 cluster centers.
We followed the same training procedure as described in
Henderson (2017) and were able to reproduce their scores on the hypernymy detection task.
A. PPMI D ETAILS
Formulation and Variants.
Typically, the probabilities used in PMIare estimated from the cooccurrence counts #( w, c ) in the corpus and lead to
PPMI (
[Equation]
, c
[Equation]
=
[Equation]
[Equation] [Equation]
(cid:19)
[Equation]
(cid:19)
[Equation]
(7) where, #( w ) =
(cid:80) c
#( w, c ) , #( c ) =
(cid:80) w #( w, c ) and | Z | = (cid:80) w (cid:80) c
#( w, c ) .
Also, it is known that PPMI is biased towards infrequent words and assigns them a higher value.
A common solution is to smoothen 8 the context probabilities by raising them to an exponent of α lying between 0 and 1 .
Levy & Goldberg (2014b) have also suggested the use of the shifted PPMI (SPPMI) matrix where the shift 9 by log( s ) acts like a prior on the probability of co-occurrence of target and context pairs.
These variants of PPMI enable us to extract better semantic associations from the co-occurrence matrix.
Finally, we have
[Equation]
Computational aspect.
We utilize the sparse matrix support of Scipy 10 for efﬁciently carrying out all the PPMI computations.
PPMI Column Normalizations.
In certain cases, when the PPMI contributions towards the partitions (or clusters) have a large variance, it can be helpful to consider the fraction of C k ’s SPPMI (Eq. 8, 9) that has been used towards a word w , instead of aggregate values used in 4.
Otherwise the process of making the histogram unit sum might misrepresent the actual underlying contribution.
We call this PPMI column normalization ( β ).
In other words, the intuition is that the normlization will balance the effect of a possible non-uniform spread in total PPMI across the clusters.
We observe that setting β to 0 .
5 or 1 help in boosting performance on the STS tasks.
The basic form of column normalization is shown in 9.
[Equation]
with
[Equation]
Another possibility while considering the normalization to have an associated parameter β that can interpolate between the above normalization and normalization with respcect to cluster size.
[Equation]
where
[Equation]
In particular, when β = 1 , we recover the equation for histograms as in 9, and β = 0 would imply normalization with respect to cluster sizes.
A. O PTIMAL T RANSPORT
Implementation aspects.
We make use of the Python Optimal Transport (POT) 11 for performing the computation of Wasserstein distances and barycenters on CPU.
For more efﬁcient GPU implementation, we built custom implementation using PyTorch.
We also implement a batched version for barycenter computation, which to the best of our knowledge has not been done in the past.
The batched barycenter computation relies on a viewing computations in the form of block-diagonal matrices.
As an example, this batched mode can compute around 200 barycenters in 0.09 seconds, where each barycenter is of 50 histograms (of size 100) and usually gives a speedup of about 10x.
Scalability.
For further scalability, an alternative is to consider stochastic optimal transport techniques (Genevay et al., 2016).
Here, the idea would be to randomly sample a subset of contexts from the distributional estimate while considering this transport.
Stability of Sinkhorn Iterations.
For all our computations involving optimal transport, we typically use λ around 0 .
1 and make use of log or median normalization as common in POT to stabilize the Sinkhorn iterations.
Also, we observe that clipping the ground metric matrix (if it exceeds a particular large threshold) also sometimes results in performance gains.
Value of p .
It has been shown in Agueh & Carlier (2011) that when the underlying space is
Euclidean and p = 2 , there exists a unique minimizer to the Wasserstein barycenter problem.
But, since we are anyways solving the regularized Wasserstein barycenter (Cuturi & Doucet, 2014) problem over here instead of the exact one, the particular value of p seems less of an issue.
Empirically in the sentence similarity experiments, we have observed p = 1 to perform better than p = 2 (by about 2-3 points).
A. C LUSTERING .
For clustering, we make use of kmcuda’s 12 efﬁcient implementation of K-Means algorithm on GPUs.
A. S OFTWARE R ELEASE
Core code and histograms.
We plan to make all our code (for all these parts) and our pre-computed histograms (for the mentioned datasets) publicly available on GitHub soon.
Standard evaluation suite for Hypernymy.
To ease the evaluation pipeline, we have collected the most common benchmark datasets and compiled the code for assessing a model’s performance on hypernymy detection or directionality into a Python package, called HypEval , which is publicly available.
This also handles OOV (out-of-vocabulary) pairs in a standardized manner and allows for efﬁcient, batched evaluation on GPU.
A. Q UALITATIVE A NALYSIS OF S ENTENCE R EPRESENTATIONS
Here, we would like to qualitatively probe the kind of results obtained when computing Wasserstein barycenter of the distributional estimates, in particular, when using CoMB to represent sentences.
To this end
[Table]
we consider a few simple sentences and ﬁnd the closest word in the vocabulary for CoMB (with respect to CMD) and contrast it to SIF with cosine distance
[Table]
Observations.
We ﬁnd that closest neighbors (see Table3 ) for CoMB consist of relatively more diverse set of words which ﬁt well in the context of given sentence.
For example, take the sentence “i love her”, where CoMB captures a wide range of contexts, for example, “i actually love her”, “i love her because ”, “i doubt her love” and more.
Also for an ambiguous sentence “he lives in europe for”, the obtained closest neighbors for CoMB include: ‘decades’, ‘masters’, ‘majority’, ‘commerce’ , etc., while with SIF the closest neighbors are mostly words similar to one of the query words.
Further, if you look at the last three sentences in the Table3, the ﬁrst closest neighbor for CoMB even acts as a good next word for the given query.
This suggests that CoMB might perform well on the task of sentence completion, but this additional evaluation is beyond the scope of this paper.
A. D ETAILED R ESULTS
Detailed results of the hypernymy detection and sentence representation experiments are listed on the following pages.
A.7. H YPERNYMY D ETECTION
[Table]
Henderson
[Table] [Table]
CMD ( α =0 .
[Table]
s =1 ) + D Hend.
[Table]
CMD ( α =0 .
15 , s =5 ) + D Hend.
[Table]
CMD ( α =0 .
15 , s =15 )
[Table] [Table]
CMD ( α =0 .
[Table]
s =1 )
+
[Table] [Table]
CMD ( α =
[Table] [Table]
[Table]
CMD ( α =
[Table]
=15 )
[Table] [Table]
CMD ( α =0 .
15 , s =1 ) + D Hend.
[Table]
CMD ( α =0 .
15 , s =5 ) + D Hend.
[Table]
A.7. S ENTENCE R EPRESENTATION
We observe empirically that the PPMI smoothing parameter α , which balances the bias of PPMI towards rare words, plays an important role.
While its ideal value would vary on each task
[Table]
we found the settings mentioned in the Table 5 to work well uniformly across the above spectrum of tasks
[Table] [Table]
[Table]
A. E FFECT OF NUMBER OF CLUSTERS
Here, we analyse the impact of number of clusters on the performance of Context Mover’s Barycenters (CoMB) for the sentence similarity experiments (c.f. Section 6).
In particular, we look at the three best performing variants (A, B, C) on the validation set (STS 16) as well as averaged across them.
We observe in Figure 3 that on average the performance signiﬁcantly improves when the number of clusters are increased until around K = 300 , and beyond that mostly plateaus ( ± 0.5).
But, as can be seen for variants B and C the performance typically continues to rise until K = 500 .
It seems that the amount of PPMI column normalization ( β = 0 .
5 vs β = 1 ) might be at play here.
Overall, going from K = 300 to K = 500 comes at the cost of increased computation time, and doesn’t lead to a substantial gain in performance.
This reﬂects the tradeoff between accuracy and efﬁciency.
Hence, we stick to K =
300 for our results on sentence similarity tasks.
Such a trend seems to be inline with the ideal case where we wouldn’t need to do any clustering and just take all possible contexts into account.
Thus, it suggests that better ways (other than clustering) to deal with this problem might further boost the performance. B Q UALITATIVE A NALYSIS OF S ENTENCE S IMILARITY
In this section, we aim to qualitatively analyse the particular examples where our method, Context
Mover’s Barycenters (CoMB), performs better or worse than the Smooth Inverse Frequency (SIF) approach from Arora et al.
B. E VALUATION P ROCEDURE
Comparing by rank.
It doesn’t make much sense to compare the raw distance values between two sentences as given by Context Mover’s Distance (CMD) for CoMB and cosine distance for SIF.
This is because the spread of distance values across sentence pairs can be quite different.
Note that the quantitative evaluation of these tasks is also carried out by Pearson/Spearman rank correlation of the predicted distances/similarities with the ground-truth scores.
Thus, in accordance with this reasoning, we compare the similarity score of a sentence pair relative to its rank based on ground-truth score (amongst the sentence pairs for that dataset).
So, the better method should rank sentence pairs closer to the ranking obtained via ground-truth scores.
[Figure]
The two sentences are completely equivalent, as they mean the same thing.
[Figure]
The two sentences are mostly equivalent, but some unimportant details differ.
[Table]
The two sentences are
[Table]
equivalent
[Table]
but
[Table]
important information differs/missing.
[Table]
[Table]
The two sentences are not equivalent, but are on the same topic.
[Table]
The two sentences are completely dissimilar.
The ground-truth scores (can be fractional) and range from 0 to 5, and the meaning implied by the integral score values can be seen in the Table 8.
In the case where different examples have the same ground-truth score, the ground-truth rank is then based on lexicographical ordering of sentences for our qualitative evaluation procedure.
(This for instance means that sentence pairs ranging from 62 to 74 would correspond to the same ground-truth score of 4.6).
The ranking is done in the descending order of sentence similarity, i.e., most similar to least similar.
Example selection criteria.
For all the examples, we compare the best variants of CoMB and SIF on those datasets.
We particularly choose those examples where there is the maximum difference in ranks according to CoMB and SIF, as they would be more indicative of where a method succeeds or fails.
Nevertheless, such a qualitative evaluation is subjective and is meant to give a better understanding of things happening under the hood.
B. E XPERIMENTS AND O BSERVATIONS
We look at examples from three datasets, namely: Images from STS15, News from STS14 and
WordNet from STS14 to get a better idea of an overall behavior.
In terms of aggregate quantitative performance, on Images and News datasets, CoMB is better than SIF, while the opposite is true for
WordNet.
These examples across the three datasets may not probably be exhaustive and are up to subjective interpretation, but hopefully will lend some indication as to where and why each method works.
B.2. T ASK : STS14, D ATASET : N EWS
We look in detail at the examples in News dataset from STS 2014 (Agirre et al., 2014).
The results of qualitative analysis on Images and WordNet datasets can be found in Section B.5.
For reference,
CoMB does a better job overall with a Pearson correlation (x100) of 64.1 versus 43.0 for SIF, as presented in Table 5.
The main observations are:
Observation 1.
Examples 1, 2, 4, 5 are sentence pairs which are equivalent in meaning (c.f.
Table 8), but typically have additional details in the predicates of the sentences.
Here, CoMB is better
[Table]
theunitedstatesandothernatomembershaverefusedratifytheamendedtreatyuntilrussiacompletelywithdrawsfrommoldovaandgeorgia
[Table]
the african union has proposed a peace
[Table]
keeping mission to aid the struggling transitional government in stabilizing so- malia , particularly after the withdrawal of ethiopian forces
[Table]
6 someaseanofﬁcialsstatedsuchstandardizationwouldbedifﬁcultduetodifferentcountries’politicalsystems.
some ofﬁcials stated the task would be difﬁcult for asean members because of varied legal and political systems .
[Table]
nicaragua
[Table]
an
[Table]
nicaragua
[Table]
not
[Table]
to ap
[Table]
the anniversary of the
[Table]
revolution .
[Table]
south korea has had a bullet train system since the
[Table]
[Table]
israel
[Table]
prolifera
[Table]
tion of
[Table]
of
[Table]
an
[Table]
china will resolutely oppose the prolifer- ation of
[Table]
[Table]
[Table]
[Table]
than SIF at ranking the pairs closer to the ground-truth ranking.
This probably suggests the averaging of word embeddings, which is the 1 st step in SIF, is not as resilient to the presence of such details than the Wasserstein barycenter of distributional estimates in CoMB.
We speculate that when having distributional estimates (where multiple senses or contexts are considered), adding details can help towards reﬁning the particular meaning implied.
Observation 2.
Let’s consider the examples 3 and 6 where SIF is better than CoMB.
These are sentence pairs which are equivalent or roughly equivalent in meanings, but with a few words substituted (typically subjects) like “judicial order” instead of “court” in example 3.
Here it seems that the substitution is adverse for CoMB while considering varied senses through the distributional estimate, in comparison to looking at the “point” meaning given by SIF.
Observation 3.
In 7, 8, and 10, each sentence pair is about a common topic, but the meaning of individual sentences is quite different.
For instance, example 8: “south korea launches new bullet train reaching 300 kph” & “south korea has had a bullet train system since the 1980s” .
Or like in example 10: “china is north korea ’ s closest ally” & “north korea is a reclusive state” .
Note that typically in these examples, the subject is same in a sentence pair, and the difference is mainly in the predicate.
Here, CoMB identiﬁes the difference and ranks them closer to the ground-truth.
Whereas,
SIF fails to understand this and ranks them as more similar (and far away) than the ground-truth.
Observation 4.
The examples 9, 11, and 12 are related sentences and differ mainly in details such as the name of the country, person, department, i.e. proper nouns.
In particular, consider example 9: “south korea and israel oppose proliferation of weapons of mass destruction and an arms race” & “china will resolutely oppose the proliferation of mass destructive weapons” .
The main difference in these examples stems from differences in the subject rather than the predicate.
CoMB considers these sentence pairs to be more similar than suggested by ground-truth.
Hence, in such scenarios where the subject (like the particular proper nouns) makes the most difference, SIF seems to be better.
B. C ONCLUSIONS FROM Q UALITATIVE E XAMPLES
Summarizing the observations from the above qualitative analysis on News dataset 13 , we conclude the following about the nature of success or failures of each method
• When the subject of the sentence is similar and main difference stems from the predicate, CoMB is the winner.
This can be seen for both the case when predicates are equivalent but described distinctly (observation 1 ) and when predicates are not equivalent (observation 3) .
• When the predicates are similar and the distinguishing factor is in the subject (or object), SIF takes the lead.
This seems to be true for both scenarios when the subject used increases or decreases the similarity as measured by CoMB, (observations 2 and 4) .
• The above two points in a way also signify where having distributional estimates can be better or worse than point estimates.
• CoMB and SIF appear to be complementary in the kind of errors they make.
Hence, combining the two is an exciting future avenue.
Lastly, it also seems worthwhile to explore having different ground metrics for CoMB and CMD (which are currently shared).
The ground metric plays a crucial role in performance and the nature of these observations.
Employing a ground metric(s) that better handles the above subtleties would be a useful research direction.
B. E FFECT OF S ENTENCE L ENGTH
In this section
[Table]
we look at the length of sentences across all the datasets in each of the STS tasks.
Average sentence length is one measure of the complexity of a particular dataset.
But looking at just sentence lengths may not give a complete picture, especially for the textual similarity tasks where there can be many words common between the sentence pairs.
The Table 10 shows the various statistics of each dataset
[Table]
with respect to the sentence lengths along with the better method on each of them (out of CoMB and SIF)
[Table]
Along with the average sentence lengths, we also measure average word overlap in the sentence pair and thus the average effective sentence length (i.e., after excluding the overlapping/common words in the sentence pair) .
For reference, we also show which out of CoMB or SIF performs better.
On STS14-Twitter, the difference in performance isn’t signiﬁcant and we thus write ‘equal’ in the corresponding cell.
Observations.
• We notice that on datasets with longer effective sentence lengths, CoMB performs better than SIF on average.
There might be other factors at play here, but if one had to pick on the axis of effective sentence length, CoMB leads over SIF 14 .
• The above statement also aligns well with the observation 1 from the qualitative analysis (c.f. Section B.2.1), that having more details can help in reﬁning the particular meaning or sense implied by CoMB.
(Effective sentence length can serve as a good proxy for indicating the amount of details.)
• It also seems to explain why both methods don’t perform well (see Table 5) on STS13FNWN, which has on average the maximum effective sentence length (of 20.4).
• To an extent, it also points towards the effect of corpora.
For instance, in a corpus such as WordNet, which has a low average sentence length and with examples typically concerned about word deﬁnitions (see Table 12), SIF seems to be better of the methods.
On the other hand, CoMB seems to be better for News (Table 9), Image captions (Table 11) or Forum.
B. A DDITIONAL Q UALITATIVE A NALYSIS B.5. T ASK : STS15, D ATASET : I MAGES
We consider the sentence pairs from Images dataset in STS15 task (Agirre et al., 2015), as presented in Table 11.
As a reminder, CoMB outperforms SIF on this dataset with a Pearson correlation (x100) of 63.1 versus 51.7, as mentioned in Table 6..
The main observations are:
[Table]
7 a man is swinging on a rope above the water .
[Table]
8 a skier wearing blue snow pants is ﬂying through the air near a jump .
a skier stands on his hands in the snow in front of a movie camera .
[Table]
9 two black and white dogs are playing together outside .
two children and a black dog are playing out in the snow .
[Table]
10 three dogs running in the dirt .
[Table]
Observation A.
Example 1 to 5 indicate pairs of sentences which are essentially equivalent in meaning, but with varying degrees of equivalence.
Here, we can see that CoMB with CMD is able to rank the similarity between these pairs quite well in comparison to SIF, even when their way of describing is different.
For instance, example 2 : “a boy waves around a sparkler” & “a young boy is twisting a sparkler around in the air” .
This points towards the beneﬁt of having multiple senses or contexts encoded through the distributional estimate in CoMB.
Observation B.
Next, in the examples 7 to 10, which consist of sentence pairs that are not equivalent but have commonalities (about the topic).
Here, SIF ranks the sentences closer to the ground-truth ranking while CoMB interprets these pairs as being more common in meaning than given by ground-truth.
This could be the consequence of comparing the various senses or contexts implied by the sentence pairs via CMD.
Take for instance, example 10, “three dogs running in the dirt” & “the yellow dog is running on the dirt road” .
Since these sentences are about the similar topic (and the major difference is in their subject), this can result in CMD considering them more similar than cosine distance.
Observation C.
For sentences which are completely dissimilar as per ground-truth, let’s look at example 11 and 12.
Consider 11, which is “a little girl and a little boy hold hands on a shiny slide”
& “a little girl in a paisley dress runs across a sandy playground”, the sentences meaning totally different things and CoMB seems to be better at ranking than SIF.
But, consider example 12: “a little girl walks on a boardwalk with blue domes in the background” & “a man going over a jump on his bike with a river in the background” .
One common theme 15 can be thought as “a person moving with something blue in the background” , which can result in CoMB ranking the sentence as more similar.
SIF also ranks it higher (at 591) than ground-truth (696), but is more closer than CoMB which ranks it at 310.
B.5. T ASK : STS14, D ATASET : W ORD N ET
Lastly, we discuss the examples and observations derived from the qualitative analysis on WordNet dataset from STS14 (Agirre et al., 2014).
This dataset is comprised of sentences which are the deﬁnitions of words/phrases, and sentence length is typically smaller than the datasets discussed before.
For reference, SIF (76.4) does better than CoMB (67.2) in terms of average Pearson correlation (x100), as mentioned in Table 5.
[Table]
cause to
[Table]
of
[Table]
3 a person or thing in the role of being a replacement for something else a person or thing that takes or can take the place of another .
[Table]
4 create something in the mind .
form a mental image of something that is not present or that is not the case .
[Table]
5 the act of surrendering an asset the act of losing or surrendering some- thing as a penalty for a mistake or fault or failure to perform etc .
[Table]
[Table]
[Table]
participate registerformallyasaparticipantormem-ber.
7 return to a prior state .
return to an original state .
[Table]
away something that is not needed .
give up what is not strictly needed .
[Table]
member
[Table]
ship .
[Table]
[Table]
[Table]
a relation
[Table]
[Table]
the act of rejecting
[Table]
Observation D. Consider examples 1 to 6 as shown in Table 12, which fall in the category of equivalent sentences but in varying degrees.
The sentence pairs essentially indicate different ways of characterizing equivalent things.
Here, CoMB is able to rank the similarity between sentences in a better manner than SIF.
Speciﬁcally, see example 2: “( cause to ) sully the good name and reputation of” & “charge falsely or with malicious intent ; attack the good name and reputation of someone” .
It seems that SIF is not able to properly handle the additional deﬁnition present in sentence 2 and ranks this pair much lower in similarity at 534 versus 235 for CoMB.
This is also in line with observationabout added details in the Section B.2.1.
Observation E.
In the examples 7 to 9, where CoMB doesn’t do well in comparison to SIF, mainly have a slight difference in the object of the sentence.
For instance, in example 9: “a person who is a member of the senate” & “a person who is a member of a partnership” .
So based on the kind of substituted word, looking at its various contexts via the distributional estimate can make it more or less similar than desired.
In such cases, using the “point” meanings of the objects seems to fare better.
This also aligns with the observations 2 and 4 in the Section B.2.1.