L EARNING E NTROPIC W ASSERSTEIN E MBEDDINGS
Anonymous authors
Paper under double-blind review
A BSTRACT
Despite their prevalence, Euclidean embeddings of data are fundamentally limited in their ability to capture latent semantic structures, which need not conform to Euclidean spatial assumptions.
Here we consider an alternative, which embeds data as discrete probability distributions in a Wasserstein space, endowed with an optimal transport metric.
Wasserstein spaces are much larger and more ﬂexible than Euclidean spaces, in that they can successfully embed a wider variety of metric structures.
We propose to exploit this ﬂexibility by learning an embedding that captures the semantic information in the Wasserstein distance between embedded distributions.
We examine empirically the representational capacity of such learned Wasserstein embeddings, showing that they can embed a wide variety of complex metric structures with smaller distortion than an equivalent
Euclidean embedding.
We also investigate an application to word embedding, demonstrating a unique advantage of Wasserstein embeddings: we can directly visualize the high-dimensional embedding, as it is a probability distribution on a low-dimensional space.
This obviates the need for dimensionality reduction techniques such as t-SNE for visualization.
I NTRODUCTION
Pre-trained embeddings form the basis for many state-of-the-art learning systems.
Word embeddings like word2vec (Mikolov et al., 2013), GloVe (Pennington et al., 2014), fastText (Bojanowski et al., 2017), and ELMo (Peters et al., 2018) are ubiquitous in natural language processing, where they are used for tasks like machine translation (Neubig et al., 2018).
Graph embeddings (Nickel et al., 2016) like node2vec (Grover & Leskovec, 2016) are used to represent knowledge graphs, and pre-trained image models are used in a wide variety of computer vision tasks.
A good embedding should represent the semantic structure of the data with high ﬁdelity, in a way that is accessible to downstream tasks.
This makes the choice of a target space for the embedding important, since different spaces can represent different types of semantic structure.
The most common choice is to embed data into Euclidean space, where distances and angles between vectors encode their levels of association (Mikolov et al., 2013; Weston et al., 2011; Kiros et al., 2014; Mirzazadeh et al., 2014).
Yet Euclidean spaces are limited in their ability to represent complex relationships between inputs, as they make restrictive assumptions about neighborhood sizes and connectivity.
This has been documented recently in the case of tree-structured data, for example, where spaces of negative curvature are required due to exponential scaling of neighborhood sizes (Nickel & Kiela, 2017; 2018).
In this paper, we embed inputs as probability distributions in a Wasserstein space.
Wasserstein spaces endow probability distributions with an optimal transport metric, which measures (in the simplest case) the distance traveled in transporting the mass in one distribution to match another.
Recent theory has shown that Wasserstein spaces are quite ﬂexible – more so than Euclidean spaces allowing a variety of other metric spaces to be embedded within them while preserving their original distance metrics.
As such, they make attractive targets for learning embeddings in machine learning, where this ﬂexibility might capture complex relationships between objects when other embeddings fail to do so
Unlike prior work on Wasserstein embeddings, which has focused on embedding into Gaussian distributions (Muzellec & Cuturi, 2018; Zhu et al., 2018), we propose to embed input data as discrete distributions, supported at a ﬁxed number of points.
In doing so we attempt to access the full ﬂexibility of Wasserstein spaces to represent a wide variety of semantic structures.
Optimal transport metrics and their gradients are costly to compute, requiring the solution of a linear program.
For efﬁciency, we use an approximation to the Wasserstein distance called the Sinkhorn divergence (Cuturi, 2013), in which the underlying optimal transport problem is regularized to render it more tractable.
While less well-characterized theoretically, with respect to embedding capacity, the Sinkhorn divergence is computed efﬁciently by a ﬁxed point iteration.
Moreover, recent work has shown that it is suitable for gradient-based optimization via automatic differentiation (Genevay et al., 2018b). To our knowledge, the current work is the ﬁrst to explore the embedding properties of the Sinkhorn divergence.
We investigate empirically two settings for Wasserstein embeddings.
First, we show their representational capacity by embedding a variety of complex networks, on which Wasserstein embeddings achieve higher ﬁdelity than both Euclidean and hyperbolic embeddings.
And second, we examine
Wasserstein word embeddings, which show retrieval performance comparable to existing methods.
One major beneﬁt of embedding into probability distributions is that the distributions can be visualized directly, unlike most embeddings, which require a dimensionality reduction step (such as t-SNE) before visualization.
We demonstrate the power of this approach by visualizing the learned word embeddings.
P RELIMINARIES 2. O PTIMAL TRANSPORT AND W ASSERSTEIN DISTANCE
The p -Wasserstein distance between probability distributions µ and ν over a metric space X is
[Equation]
[Equation] where the inﬁmum is taken over transport plans π that distribute the mass in µ to match that in ν , with the p -th power of the ground metric d ( x 1 , x 2 ) on X giving the cost of moving a unit of mass from support point x 1 ∈ X underlying distribution µ to point x 2 ∈ X underlying ν .
The
Wasserstein distance is the cost of the optimal transport plan matching µ and ν (Villani, 2003).
In this paper, we are concerned with discrete distributions supported on ﬁnite sets of points in R n :
[Equation]
Here, u and v are vectors of nonnegative weights summing to 1 , and { x ( i ) } Mi =1 , { y ( i ) } Ni =1 ⊂ R n are the support points .
In this case, the optimal transport plan π matching µ and ν in equationbecomes discrete as well, supported on the product of the supports.
Deﬁne D ∈ R M × N + to be the matrix of pairwise ground metric distances, with D ij
=
(cid:107) x
( i ) − y
( j ) (cid:107) p .
Then, for our discrete distributions, equation 1 is equivalent to solving the following:
[Equation]
subject to
[Equation]
with T ij giving the transported mass between x i and y j .
The power D p is taken elementwise.
2. S INKHORN DIVERGENCE
Equation 3 is a linear program that can be challenging to solve in practice.
To improve efﬁciency of numerical optimal transport, recent learning algorithms use an entropic regularizer proposed by
Cuturi (2013).
The resulting Sinkhorn divergence solves a slightly modiﬁed version of equation 3:
[Equation]
where log( · ) is applied elementwise, ◦ is a Hadamard (elementwise) product, and λ ≥ 0 is a regularization parameter.
For λ > 0 , the optimal solution takes the form T ∗ =
∆( r ) exp ( − D p / λ ) ∆( c ) , for new vector variables r and c , where ∆( · ) puts a vector along the diagonal of a square matrix.
Hence, rather than optimizing over matrices T , one can optimize over vectors r and c , reducing the size of the problem to M + N .
This can be solved via matrix rebalancing starting from an initial matrix
K := exp( − D p λ ) and alternatively projecting onto the marginal constraints until convergence:
[Equation]
Here, ./ denotes elementwise division for vectors.
Beyond simplicity of implementation, equation 5 has an additional advantage for machine learning applications: The steps of this algorithm are easily differentiable.
With this observation in mind,
Genevay et al.
(2018b) incorporate entropic transport into learning pipelines by applying automatic differentiation (back propagation) to a ﬁxed number of Sinkhorn iterations.
2. W HAT CAN WE EMBED IN THEORY ?
Given two metric spaces A and B , an embedding of A into B is a map φ : A → B that preserves distances, in the sense that the distortion is small:
[Equation]
for some uniform constants L > 0 and C ≥ 1 .
The distortion of the embedding φ is the smallest C such that equation 6 holds.
One can characterize how “large” a space is (its representational capacity ) by the spaces that successfully embed into it with low distortion.
In practical terms, this capacity determines the types of data (and relationships between them) that can be well-represented in the embedding space.
R n with the Euclidean metric, for example, embeds into the L 1 metric with low distortion, while the reverse is not true (Deza & Laurent, 2009).
We do not expect Manhattan-structured data to be well-represented in Euclidean space, no matter how clever the mapping.
Wasserstein spaces are very large: Many spaces can embed into Wasserstein spaces with low distortion, even when the converse is not true.
W p
( A ) , for A an arbitrary metric space, embeds any product space A n , for example (Kloeckner, 2010), via discrete distributions supported at n points.
Even more generally, certain Wasserstein spaces are universal , in the sense that they can embed arbitrary metrics on ﬁnite spaces .
W 1 ( (cid:96) 1 ) is one such space Bourgain (1986), and it is still an open problem to determine if W 1 ( R k ) is universal for any k < + ∞ .
Recently it has been shown that every ﬁnite metric space embeds the 1 p power of its metric into W p
( R 3 ) , p > 1 , with vanishing distortion (Andoni et al., 2015).
A hopeful interpretation suggests that W 1 ( R 3 ) may be a plausible target space for arbitrary metrics on symbolic data, with a ﬁnite set of symbols.
The authors are not aware of any similar universality results for L p or hyperbolic spaces, for example.
The reverse direction, embedding Wasserstein spaces into others, is particularly well-studied in the case of discrete distributions.
Theoretical results in this domain are motivated by interest in efﬁcient algorithms for approximating Wasserstein distances by embedding them into spaces with more easily-computed metrics.
In this direction, low-distortion embeddings are not easy to ﬁnd.
W 2 ( R 3 ) , for example, is known not to embed into L 1 space (Andoni et al., 2016).
Some positive results exist, nevertheless.
For a Euclidean ground metric, for example, the 1 -Wasserstein distance can be approximated in a wavelet domain (Shirdhonkar & Jacobs, 2008) or by high-dimensional embedding into L 1 (Indyk & Thaper, 2003).
In § 4, we investigate empirically the embedding capacity of Wasserstein spaces, by attempting to learn low-distortion embeddings for a variety of input spaces.
Note that, for efﬁciency, we replace the Wasserstein distance by its entropy-regularized counterpart, the Sinkhorn divergence (Section 2.2).
The embedding capacity of Sinkhorn divergences is previously unstudied, to our knowledge, except in the weak sense that the approximation error with respect to the Wasserstein distance vanishes with the regularizer taken to zero (Carlier et al., 2017; Genevay et al., 2018a).
2. R ELATED WORK
While learning to embed into a vector space has a long history, there is a recent tendency in the representation learning community to generalize from individual points to more complex structures and spaces like probability distributions (Vilnis & McCallum, 2015; Athiwaratkun & Wilson, 2018),
Euclidean norm balls (Mirzazadeh et al., 2015; Mirzazadeh, 2017), Poincar´e balls (Nickel & Kiela, 2017), and Lorentz models (Nickel & Kiela, 2018).
From a modeling perspective, generalization to other spaces helps models carry information like uncertainty (Vilnis & McCallum, 2015; Bojchevski & G¨unnemann, 2018) to the objects being embedded (Vilnis & McCallum, 2015).
More importantly, alternative embedding spaces have the capacity to encode relations like inclusion, exclusion, hierarchy, and ordering (Mirzazadeh et al., 2015; Vendrov et al., 2015; Athiwaratkun & Wilson, 2018).
Our work share the same themes, since we take point clouds as the embedding targets.
The distance or discrepancy measure between points in an embedding space is another major deﬁning factor for a representation learning model.
For points, the most typical choice is to use an L p norm.
When embedding into the set of histograms, discrepancy typically is evaluated using KL divergence (Kullback & Leibler, 1951).
KL divergence between two distributions can be problematic, however.
In particular, the behavior of KL divergence relies heavily on distributions having common non-empty support; once the support sets are disjoint, regardless of how close or far the support points lie on the underlying space, the KL divergence remains the same.
As an alternative discrepancy measure, transport distances (Villani, 2008; Peyr´e & Cuturi, 2017; Solomon, 2018) do not suffer from the mentioned problems.
Hence, models based on optimal transport are gaining popularity in machine learning; see (Rubner et al., 1998; Courty et al., 2014; Frogner et al., 2015; Kusner et al., 2015; Arjovsky et al., 2017; Genevay et al., 2018b; Claici et al., 2018) for some examples.
Learned embeddings into Wasserstein spaces do not have long precedence.
A recent line of research proposes embedding into parameterized sets of distributions (Muzellec & Cuturi, 2018; Zhu et al., 2018).
While restricting to smaller sets facilitates use of closed-form expressions for transport distances, restriction to Gaussian distributions leads to a less expressive representation space.
It is noteworthy that Courty et al.
(2018) study embedding from Wasserstein into Euclidean space.
In contrast, we learn to embed into the space of discrete probability distributions endowed with the Wasserstein distance, well-known to be dense in W 2 as the number of support points increases (Kloeckner, 2012; Brancolini et al., 2009).
L EARNING W ASSERSTEIN EMBEDDINGS 3. T HE LEARNING PROBLEM
The learning task we consider is that of recovering a pairwise distance or similarity relationship that is partially or fully observed.
We are given a collection of objects C —these can be words, symbols, images, or any other data—as well as samples (cid:8)(cid:0) u
( i ) , v ( i ) , r ( u ( i ) , v ( i ) )
(cid:1)(cid:9) of a target relationship r : C × C
→ R that tells us the degree to which pairs of objects are related.
Our objective is to ﬁnd a map φ : C → W p ( X ) such that the relationship r ( u, v ) can be recovered from the Wasserstein distance between φ ( u ) and φ ( v ) , for any u, v ∈ C .
Examples include:
M ETRIC EMBEDDING : r is a distance metric
and we want W p ( φ ( u ) , φ ( v ))
≈ r ( u v ) for all u, v ∈ C . 2.
G RAPH EMBEDDING :
C contains the vertices of a graph and r : C×C → { 0 , 1 } is the adjacency relation; we would like the neighborhood of each φ ( u ) in W p to coincide with graph adjacency.
W ORD EMBEDDING : C contains individual words and r is a semantic similarity between words.
We want distances in W p to predict this semantic similarity.
Although the details of each task require some adjustment to the learning architecture, our basic representation and training procedure detailed below applies to all three examples.
3. O PTIMIZATION
Given a set of training samples S = (cid:8)(cid:0) u
( i ) , v ( i ) ,
r ( i ) (cid:1)(cid:9) N i =1 ⊂ C × C × R , we want to learn a map φ : C → W p ( X ) .
We must address two issues.
First we must deﬁne the range of our map φ .
The whole of W p ( X ) is inﬁnite-dimensional, and for a tractable problem we need a ﬁnite-dimensional output.
We restrict ourselves to discrete distributions with an a priori ﬁxed number of support points M , reducing optimal transport to the linear program in equation 3.
Such a distribution is parameterized by the locations of its support points { x ( j ) } Mj =1 , forming a point cloud in the ground metric space X .
For simplicity, we restrict to uniform weights u , v ∝ 1 , although it is certainly possible to optimize simultaneously over weights and locations. As noted in (Brancolini et al., 2009; Kloeckner, 2012; Claici et al., 2018), however, when constructing a discrete M -point approximation to a ﬁxed target distribution, allowing non-uniform weights does not improve the asymptotic approximation error.
The second issue is that, as noted in § 2.2, exact computation of W p in general is costly, requiring the solution of a linear program.
As in (Genevay et al., 2018b), we instead replace W p with the Sinkhorn divergence W λp , which is solvable by a simple ﬁxed-point iteration introduced in § 2.2, equation 5.
Learning then takes the form of empirical loss minimization,
[Equation]
over a hypothesis space of maps H .
The loss L is problem-speciﬁc and scores the coincidence between the (regularized) Wasserstein distance W λp and the target relationship r , evaluated at the pair (cid:0) u ( i ) , v ( i ) (cid:1) .
As mentioned in § 2.2, gradients are available from automatic differentiation of the Sinkhorn procedure, and hence with a suitable loss function the learning objective 7 can be optimized by standard gradient-based methods.
In the experiments that follow, we use the Adam optimizer.
E MPIRICAL STUDY 4. R EPRESENTATIONAL CAPACITY : EMBEDDING COMPLEX NETWORKS
We ﬁrst demonstrate the representational power of learned Wasserstein embeddings.
As discussed in § 2.3, theory suggests that Wasserstein spaces are quite ﬂexible, in that they can embed a wide variety of metrics with low distortion.
We show that this is true in practice as well.
To generate a variety of metrics to embed, we take networks with various patterns of connectivity and compute the shortest path distances between vertices.
The collection of vertices for each network serves as the input space C for our embedding, and our goal is to learn a map φ : C → W p ( R k ) such that the Wasserstein distance W p ( φ ( u ) , φ ( v )) matches as closely as possible the shortest path distance between vertices u and v , for all pairs of vertices.
We learn a minimum distortion embedding : given a fully observed distance metric d C : C × C → R in the input space, we minimize the mean distortion:
[Equation]
φ is parameterized as in § 3.2, directly specifying the support points of the output distribution.
We examine the performance of Wasserstein embedding using both random networks and real networks.
The random networks in particular allow us systematically to test robustness of the Wasserstein embedding to particular properties of the metric we are attempting to embed.
Note that these experiments do not explore generalization performance: we are purely concerned with the representational capacity of the learned Wasserstein embeddings.
For random networks, we use three standard generative models: Barab´asi–Albert (Albert &
Barab´asi, 2002), Watts–Strogatz (Watts & Strogatz, 1998), and the stochastic block model (Holland et al., 1983).
Random scale-free networks are generated from the Barab´asi–Albert model, and possess the property that distances are on average much shorter than in a Euclidean spatial graph, scaling like the log of the number of vertices.
Random small-world networks are generated from the Watts–Strogatz model; in addition to log-scaling of the average path length, Watts–
Strogatz graphs also show clustering of vertices into distinct neighborhoods.
Random communitystructured networks are generated from the stochastic block model, which places vertices within densely-connected communities, with only sparse connections between the different communities.
[Figure]
We additionally generate random trees by choosing a random number of children 2 for each node, progressing in breadth-ﬁrst order until a speciﬁed total number of nodes is reached.
In all cases, we generate networks with 128 vertices.
We compare against two baselines, trained using the same distortion criterion and optimization method: Euclidean embeddings , and hyperbolic embeddings .
Euclidean embeddings we expect to struggle with all of the chosen graph types, as they are limited to representing spatial relationships with zero curvature.
Hyperbolic embeddings model tree-structured metrics, as they capture the exponential scaling of the graph neighborhoods, and have been suggested for a variety of other graph families as well (Zhao et al., 2011).
Figure 1 shows the result of embedding random networks 3 .
As the total embedding dimension increases, the distortion decreases for all of the embedding methods.
Importantly, the Wasserstein embeddings achieve lower distortion than both the Euclidean and hyperbolic embeddings, establishing their ﬂexibility under the varying conditions represented by the different network models.
In some cases, the Wasserstein distortion continues to decrease long after the other embeddings have saturated their capacity.
Note that, as expected, the hyperbolic space signiﬁcantly outperforms both
Euclidean and Wasserstein on tree-structured metrics.
Note also that we test the R 2 , R 3 , and R 4 ground metric spaces.
For all of the random networks we examined, the performance between R 3 and R 4 ground metrics is nearly indistinguishable.
This is consistent with theoretical results ( § 2.3) that suggest that R 3 is sufﬁcient to embed a wide variety of metrics. [Figure]
We also examine fragments of real networks : an ArXiv co-authorship network, an Amazon product co-purchasing network, and a Google web graph (Leskovec & Krevl, 2014).
For each graph fragment, we choose uniformly at random a starting vertex, then extract the subgraph on 128 vertices taken in breadth-ﬁrst order from that starting vertex.
The results of this experiment are shown in Figure 2.
Again we see that the Wasserstein embeddings achieve lower distortion than either Euclidean or hyperbolic embeddings.
4. WORD 2 CLOUD : W ASSERSTEIN WORD EMBEDDINGS
In this section, we focus on embedding of words to point clouds.
In a sentence s =
( x 0 , . . .
, x n ) , a word x i is associated with word x j if x j is in the context of x i .
Here, context is deﬁned as a symmetric window around x i and the association is shown with labels r , that is, the label r x i , x j =if and only if | i − j | ≤ l where l is the window size.
For word embedding, we used a contrastive loss function (Hadsell et al., 2006):
[Equation]
which tries to embed two words x i , x j close to each other in terms of Wasserstein distance (here
W λ 1 ) if they are in the context of each other and at least margin m away otherwise; this approach is similar to that suggested by Mikolov et al.
(2013), up to the loss and distance functions.
We used a Siamese architecture (Bromley et al., 1993) for our model, with negative sampling (again like Mikolov et al.
for selecting words out of the context.
The network architecture in each branch consists of a linear layer with 64 nodes followed by our point cloud embedding layer.
The
Wasserstein distance layer connects the two branches of the Siamese network.
Training dataset is Text8 4 which is of moderate size but commonly used as a language modeling benchmark and consists of a text corpus with 17 M tokens from Wikipedia.
We used a vocabulary subset of size 8000 and a window size of l = 2 (i.e., 2 words on each side), λ = 0 .
05 , number of epochs of 3 , negative sampling rate of 1 and Adam (Kingma & Ba, 2014) for optimization.
Given a ﬁxed number of parameters, ﬁrst we study the effect of dimensionality of point clouds in the semantic neighborhood captured by the embeddings.
Since the total number of parameters is ﬁxed, increasing dimensionality means losing some points.
Table 1 shows the 5 nearest neighbors using the distance measure trained shown in the ﬁrst block.
Here, the total number of parameters used is in { 63 , 64 } .
Interestingly, it is more effective to use a budget of 64 parameters in 16 -point, 4dimensional cloud than in a 32 -point, 2 -dimensional cloud.
It is notable that changing the structure of embedding target in this way, can directly improve the quality of the representation learned.
Next we evaluate these models on a number of benchmark similarity tasks from (Faruqui & Dyer, 2014) to compare the quality of the embeddings in a retrieval task when we change the dimensionality of the point clouds with ﬁxed total number of parameters.
Results are shown in Table 2.
Here, for consistency, we have picked the latent layer before our cloud layer and considered it as the embedding to be compared.
The results of our method appear in the middle block of Table 2.
The gradual improvement with increasing the dimensionality of the point clouds.
The right block in Table 2 shows baselines: Respectively RNN(80D) (Kombrink et al., 2011), Metaoptimize (50D) [Table]
[Table] (Turian et al., 2010), SENNA (50D) (Collobert, 2011)
Global Context (50D) (Huang et al., 2012) and word2vec (80D) (Mikolov et al., 2013).
The reported performance measure is based on (Faruqui &
Dyer, 2014) computing the correlation with ground-truth of the rankings.
All correlations are based on (Faruqui & Dyer, 2014).
Note that there are many ways to improve the performance: increasing the vocabulary/window size/number of epochs/negative sampling rate, using larger texts, and accelerating performance.
We defer this tuning to future work focused speciﬁcally on NLP.
4.2. D IRECT , INTERPRETABLE VISUALIZATION OF HIGH - DIMENSIONAL EMBEDDINGS
Wasserstein embeddings over low-dimensional ground metric spaces have a unique property: We can directly visualize the embedding , which is a point cloud in the low-dimensional ground space.
This is not true for most existing embedding methods, which rely on dimensionality reduction techniques such as t-SNE for visualizing the embedded data.
Whereas dimensionality reduction only approximately captures closeness of points in the embedding space, with Wasserstein embeddings we can see the exact embedding of each input, by visualizing the point cloud.
We demonstrate this capability by visualizing the learned word representations.
Importantly, the point clouds strongly cluster, which leads to apparent, distinct modes in their density.
We therefore use kernel density estimation to visualize their densities.
In Figure 3a, we visualize three distinct words, thresholding each density at a low value and visualizing its upper level set, which reveals the modes.
These level sets are overlaid, with each color in the the ﬁgure corresponding to a distinct embedded word.
The density for each word is depicted by the opacity of the color within each level set.
We ﬁrst note that it is easy to visualize in aggregate multiple sets of words, by assigning all words within a set a single color.
This immediately reveals how well-separated are the sets.
This is shown in Figure 3b: as expected, there is some overlap between military and political terms, while names of sports are separated from the rest.
Looking at the embeddings in more detail, we can start to dissect relationships (and confusion) between different sets of words.
We observe that each word tends to concentrate its mass in two or more distinct regions.
This multimodal nature of the density allows for multifaceted relationships [Figure] between words, as a word can partially overlap with many distinct groups of words simultaneously.
Figure 3c shows the embedding for a word that has multiple distinct meanings ( kind ), alongside synonyms for both senses of the word ( nice, friendly, type ).
We see that kind has two primary modes, which overlap separately with friendly and type .
nice is included to show a failure of the embedding to capture the full semantics – Figure 3d shows that the network has in fact learned that nice is a city in France, while ignoring its second interpretation.
This demonstrates the potential of this visualization for debugging our network, by identifying and attributing an error.
D ISCUSSION AND CONCLUSION
Several characteristics determine the value and effectiveness of an embedding space for representation learning.
The space must be large enough to embed a variety of metrics, while admitting a mathematical description compatible with learning algorithms; additional features, including direct interpretability, make it easier to understand, analyze, and potentially debug the output of a representation learning procedure.
Based on their theoretical properties, Wasserstein spaces are strong candidates for representing complex semantic structures, when the capacity of Euclidean space does not sufﬁce.
Empirically, entropy-regularized Wasserstein distances are effective for embedding a wide variety of semantic structures, while enabling direct visualization of the embedding.
Our work suggests several directions for additional research.
Beyond simple extensions like weighting points in the point cloud, one observation is that we can lift nearly any representation space X to distributions over that space W ( X ) represented as point clouds; in this paper we focused on the case X = R n .
Since X embeds within W ( X ) using δ -functions, this might be viewed as a general “lifting” procedure increasing the capacity of a representation.
We can also consider other tasks, such as co-embedding of different modalities into the same transport space.
Additionally, our empirical results suggest that theoretical study of the embedding capacity of Sinkhorn divergences may be proﬁtable. And, following recent work on computing geodesics in Wasserstein space (Seguy & Cuturi, 2015), it may be interesting to invert the learned mappings and use them for interpolation.