A Probabilistic Graph Coupling View of Dimension Reduction
Abstract
Most popular dimension reduction (DR) methods like t-SNE and UMAP are based on minimizing a cost between input and latent pairwise similarities.
Though widely used, these approaches lack clear probabilistic foundations to enable a full understanding of their properties and limitations.
To that extent, we introduce a unifying statistical framework based on the coupling of hidden graphs using cross entropy.
These graphs induce a Markov random ﬁeld dependency structure among the observations in both input and latent spaces.
We show that existing pairwise similarity DR methods can be retrieved from our framework with particular choices of priors for the graphs.
Moreover this reveals that these methods relying on shift-invariant kernels su  er from a statistical degeneracy that explains poor performances in conserving coarse-grain dependencies.
New links are drawn with PCA which appears as a non-degenerate graph coupling model.
Introduction
Dimensionality reduction (DR) is of central importance when dealing with high-dimensional data [15].
It mitigates the curse of dimensionality, allowing for greater statistical ﬂexibility and less computational complexity.
DR also enables visualization that can be of great practical interest for understanding and interpreting the structure of large datasets.
Most seminal approaches include Principal Component Analysis (PCA) [31], multidimensional scaling [23] and more broadly kernel eigenmaps methods such as Isomap [4], Laplacian eigenmaps [5] and di  usion maps [12].
These methods share the deﬁnition of a pairwise similarity kernel that assigns a high value to close neighbors and the resolution of a spectral problem.
They are well understood and uniﬁed in the kernel PCA framework [16]
In the past decade, the ﬁeld has witnessed a major shift with the emergence of a new class of methods.
They are also based on pairwise similarities but these are not converted into inner products.
Instead, they deﬁne pairwise similarity functions in both input and latent spaces and optimize a cost between the two.
Among such methods, the Stochastic Neighbor
Embedding (SNE) algorithm
[18], its heavy-tailed symmetrized version t-SNE [38] or more recent approaches like LargeVis [36] and UMAP [30] are arguably the most used in practice.
These will be referred to as SNE-like or neighbor embedding methods in what follows.
They are increasingly popular and now considered as the state-of-art techniques in many ﬁelds
Their popularity is mainly due to their exceptional ability to preserve local structure, i.e. close points in the input space have close embeddings, as shown empirically [40].
They also demonstrate impressive performances in identifying clusters [3, 29].
However this is done at the expense of global structure, that these methods struggle in preserving [41, 11] i.e. the relative large-scale distances between embedded points do not necessarily correspond to the original ones.
Due to a lack of clear probabilistic foundations, these properties remain mostly empirical.
This gap between theory and practice is detrimental as practitioners may rely on strategies that are not optimal for their use case.
While recent software developments are making these methods more scalable [8, 33, 28] and further expanding their use, the need for a well-established probabilistic framework is becoming more prominent.
In this work we deﬁne the generative probabilistic model that encompasses current embedding methods, while establishing new links with the well-established PCA model.
Outline.
Consider X =
( X 1 , ..., X n ) € œ R n ◊ p , an input dataset that consists of n vectors of dimension p .
Our task is to embed X in a lower dimensional space of dimension q < p (typically q = 2 for visualization), and we denote by Z
= ( Z 1 , ..., Z n ) € œ R n ◊ q the unknown embeddings.
The rationale of our framework is to suppose that the observations X and Z are structured by two latent graphs with W X and W Z standing for their n -square weight matrices.
As the goal of DR is to preserve the input’s structure in the latent space, we propose to ﬁnd the best low-dimensional representation Z of X such that W X and W Z are close.
To build a ﬂexible and robust probabilistic framework, we consider random graphs distributed according to some predeﬁned prior distributions.
Our objective is to match the posterior distributions of W X and W Z .
Note that as they share the same dimensionality the latter graphs can be easily compared unlike X and Z .
The coupling is done with a cross entropy criterion, the minimization of which will be referred to as graph coupling.
In this work, our main contributions are as follows.
• We show that SNE, t-SNE, LargeVis and UMAP are all instances of graph coupling and characterized by di  erent choices of prior for discrete latent structuring graphs (section 3).
We demonstrate that such graphs essentially capture conditional independencies among rows through a pairwise Markov Random Field (MRF) model which construction can be found in section 2.
• We uncover the intrinsic probabilistic property explaining why such methods perform poorly on conserving the large scale structure of the data as a consequence of a degeneracy of the MRF when shift invariant kernels are used (theorem 1).
Such degeneracy induces the loss of the relative positions of clusters corresponding to the connected components of the posterior latent graphs which distributions are identiﬁed (proposition 1).
These ﬁndings are highlighted by a new initialization of the embeddings that is empirically tested (section 4).
• We show that for Gaussian MRFs, when adapting graph coupling to precision matrices with suitable priors, PCA appears as a natural extension of the coupling problem in its continuous version (theorem 2).
Such model does not su  er from the aforementioned degeneracy hence preserves the large-scale structure.
Shift-Invariant Pairwise MRF to Model Row Dependencies
We start by deﬁning the distribution of the observations given a graph.
The latter takes the form of a pairwise MRF model which as we show is improper ( i.e. not integrable on R n ◊ p ) when shift-invariant kernels are used.
We consider a ﬁxed directed graph W œ S W where:
[Equation]
* Throughout, ( E, B ( E ) , ⁄ E ) denotes a measure space where B ( E ) is the Borel ‡ -algebra on E and ⁄
E is the Lebesgue measure on E .
2. Graph Laplacian Null Space
A central element in our construction is the graph Laplacian linear map, deﬁned as follows, where S n + ( R ) is the set of positive semideﬁnite matrices.
Deﬁnition
The graph Laplacian operator is the map L : R n ◊ n + æ S n + ( R ) such that for ( i, j ) œ [ n ] 2 ,
L ( W ) ij = ; ≠ W ij if i ” =
j q k œ [ n ] W ik otherwise .
With an abuse of notation, let L =
L ( W ) where W = W + W € .
Let ( C 1 , ..., C R ) be a partition of [ n ] ( i.e. the set { 1 , 2 , ..., n } ) corresponding to the connected components (CCs) of W .
As well known in spectral graph theory [9], the null space of L is spanned by the orthonormal vectors { U r } r œ
[ R ] such that for r œ [ R ] , U r = n ≠ 1 / 2 r 11 i œ C
r 2 i œ [ n ] with n r = Card( C r ) .
By the spectral theorem, U [ R ] can be completed such that L
= U
⇤ U > where U
= ( U 1 , ..., U n ) is orthogonal and ⇤ =
diag(( ⁄ i ) i œ [ n ] ) with 0 = {+⁄ 
+}
= ...
= ⁄
R < ⁄ R {++
 Æ+} ... Æ ⁄ n .
In the following, the data is split into two parts: X M , the orthogonal projection of X on S M =
(ker L ) ¢ R p , and X C , the projection on S C =
(ker L ) ‹ ¢ R p .
For i œ [ n ] , X M ,i = q [Equation] ] n ≠
[Equation]
i [Equation] r q ¸ œ C r X ¸ hence X M stands for the empirical means of X on CCs, thus modelling the CC positions, while X C = X
≠ X M is CC-wise centered, thus modeling the relative positions of the nodes within CCs.
We now introduce the probability distribution of these variables.
2. Pairwise MRF and Shift-Invariances
In this work, the dependency structure among rows of the data is governed by a graph.
The strength of the connection between two nodes is given by a symmetric function k : R p æ R + .
We consider the following pairwise MRF unnormalized density function:
[Equation]
As we will see shortly, the above is at the heart of DR methods based on pairwise similarities.
Note that as k measures the similarity between couples of samples, f k will take high values if the rows of X vary smoothly on the graph W .
Thus we can expect X i and X j to be close if there is an edge between node i and node j in W .
A key remark is that f k is kept invariant by translating X M .
Namely for all X œ R n ◊ p , f k ( X , W )
=
f k ( X C , W ) .
This invariance results in f k ( · , W ) being non integrable on R n ◊ p , as we see with the following example.
Gaussian kernel.
For a positive deﬁnite matrix ⌃ œ S n ++ ( R ) , consider the Gaussian kernel k : x ‘æ e [-− 
 ∥-]≠ 12 Î x Î 2 ⌃ where ⌃ stands for the covariance among columns.
One has:
[Equation]
by property of the graph Laplacian (deﬁnition 1).
In this case, it is clear that due to the rank deﬁciency of L , f k ( · , W ) is only ⁄ S C -integrable.
In general DR settings one does not want to rely on Gaussian kernels only.
A striking example is the use of the Student kernel in t-SNE [38].
Heavy-tailed kernels appear useful when the dimension of the embeddings is smaller than the intrinsic dimension of the data [20].
Our contribution provides ﬂexibility by extending the previous result to a large class of kernels, as stated in the following theorem.
Theorem 1 If k is ⁄ R p -integrable and bounded above ⁄ R p -almost everywhere then f k ( · , W ) is ⁄ S C -integrable.
We refer to appendix A.1 for the proof.
We can now deﬁne a distribution on ( S C , B ( S C )) , where C k ( W ) =
[Equation]
Remark 1 Kernels may have node-speciﬁc bandwidths ⌧ , set during a pre-processing step, giving f k ( X , W ) =
r ( i,j ) k
(( X i ≠ X j ) / · i ) W ij .
Note that such bandwidth does not a  ect the degeneracy of the distribution and theorem 1 still holds.
Between-Rows Dependency Structure.
By symmetry of k , reindexing gives: f k ( X , W ) =
r j œ [ n ] r i œ [ j ] k ( X i ≠ X j ) W ij .
Hence distribution (3) boils down to a pairwise MRF model [10] with respect to the undirected graph W , C k playing the role of the partition function.
Note that since f k (Equation 1) trivially factorize according to the cliques of W , the Hammersley-Cli  ord theorem ensures that the rows of X C satisfy the local and global Markov properties with respect to W .
2. Uninformative Model for CC-wise Means
We showed that the MRF (1) is only integrable on S C , the deﬁnition of which depends on the connectivity structure of W .
As we now demonstrate, the latter MRF can be seen as a limit of proper distributions on R n ◊ p , see e.g. [35] for a similar construction in the Gaussian case.
We introduce the Borel function f Á ( · , W ): R n ◊ p æ R + for Á > 0 such that for all X œ R n ◊ p , f Á ( X , W ) =
f Á ( X M , W ) .
To allow f Á to become arbitrarily non-informative, we assume that for all W œ S W , f Á ( · , W ) is ⁄ S M -integrable for all Á œ R ú + and f Á ( · , W ) ≠≠≠æ Á æ 0 1 almost everywhere.
We now deﬁne the conditional distribution on ( S M , B ( S M )) as follows:
[Equation]
where C Á ( W ) = s f Á ( · , W ) d ⁄ S M .
With this at hand, the joint conditional is deﬁned as the product measure of (3) and (4) over the row axis, the integrability of which is ensured by the Fubini-Tonelli theorem.
In the following we will use the compact notation
C Á k ( W ) =
C k ( W ) C Á ( W ) for the joint normalizing constant.
Remark
At the limit Á æ 0 the above construction amounts to setting an inﬁnite variance on the distribution of the empirical means of X on CCs, thus loosing the inter-CC structure.
As an illustration, one can structure the CCs’ relative positions according to a Gaussian model with positive deﬁnite precision Á ⇥ œ S R ++ ( R ) , as it amounts to choosing f Á : X æ exp
[Equation] " such that: vec( X M ) | ⇥ ≥ N 0 , ! [Equation] U [
[Equation]
R ] ⇥ U €
[Equation] R ] " [Equation] ⌃ 2 where ¢ denotes the Kronecker product.
Graph Coupling as a Uniﬁed Objective for Pairwise Similarity Methods
In this section, we show that neighbor embedding methods can be recovered in the presented framework.
They are obtained, for particular choices of graph priors, at the limit Á æwhen f Á becomes non informative and the CCs’ relative positions are lost.
We now turn to the priors for W .
Our methodology is similar to that of constructing conjugate priors for distributions in the exponential family [39], notably we insert the cumulant function C Á k ( i.e. normalizing constant of the conditional) as a multivariate term of the prior.
We consider di  erent forms: binary ( B ), unitary out-degree ( D ) and n -edges ( E ), relying on an additional term (  ) to constraint the topology of the graph.
For a matrix
A , A i + denotes q j A ij and A ++ denotes q ij A ij .
In the following, ⇡ plays the role of the edge’s prior.
The latter can be leveraged to incorporate some additional information about the dependency structure, for instance when a network is observed [26].
Deﬁnition 2 Let ⇡ œ R n ◊ n + , Á œ R + , – œ R , k satisﬁes the assumptions of theorem 1 and
P œ { B, D, E } .
For W œ S W we introduce:
[Equation]
where  B ( W ) =
r ij 11 W ij Æ 1 ,  D ( W ) =
{+r i 
+}
W i + =1 and  E ( W ) =
11 W ++ = n r ij ( W ij !)
≠ 1 .
When – = 0 , the above no longer depends on Á and k .
We will use the compact notation
P P ( W ; ⇡ ) = P Á P ,k ( W ; ⇡ , 0) .
Note that by W ≥ P P ( · ; ⇡ ) we have the following simple
Bernoulli ( B ) and multinomial ( M ) distributions, where matrix or vector division is to be understood as element-wise.
• If P = B , ’ ( i, j ) œ [ n ] 2 , W ij ‹‹ ≥ B ( ﬁ ij / (1 + ﬁ ij )) .
• If P = D , ’ i œ [ n ] , W i ‹‹ ≥ M (1 , ⇡ i / ﬁ i + ) .
• If P = E , W ≥ M ( n, ⇡ / ﬁ ++ ) .
We now show that the posterior distribution of the graph given the observations takes a simple form when the distribution of CC empirical means X M di  uses i.e. when Á æ 0 (a proof of the following result can be found in appendix A.2).
In the following, § stands for the Hadamard product and D for the convergence in distribution.
Proposition 1 Let ⇡ œ R n ◊ n + , k satisﬁes the assumptions of theorem 1 with K X =
( k ( X i ≠
X j ))
( i,j ) œ [ n ] 2 and P œ { B, D, E } .
If W Á ≥ P Á P ,k ( · ; ⇡ , 1) then
[Equation]
Remark {+
+} For all W œ S W , C Á ( W ) diverges as Á æ 0 , hence the graph prior (deﬁnition 2) is improper at the limit.
This compensates for the uninformative di  use conditional and allows to retrieve a well-deﬁned tractable posterior limit.
3. Retrieving Well Known DR Methods
We now provide a uniﬁed view of neighbor embedding objectives as a coupling between graph posterior distributions.
To that extent we derive the cross entropy associated with the various graph priors at hand.
In what follows, k x and k z satisfy the assumptions of theoremand we denote by K X and K Z the associated kernel matrices on X and Z respectively.
For both graph priors we consider the parameters ⇡ = 1 and – = 1 .
For ( P X , P Z ) œ { B, D, E } 2 , we introduce the cross entropy between the limit posteriors at Á
[Equation]
deﬁning a coupling criterion to be optimized with respect to embedding coordinates Z .
We now go through each couple ( P X , P Z ) such that supp ( P P X ) µ supp ( P P Z ) for the cross-entropy to be deﬁned.
SNE.
When P X = P Z = D , the probability of the limit posterior graphs factorizes over the nodes and the cross-entropy between limit posteriors takes the form of the objective of
SNE [17], where for i œ [ n ] , P Di = K X ,i /K
X ,i + and Q Di
= K Z ,i /K
Z ,i + ,
[Equation]
Symmetric-SNE.
Choosing P X
= D and P Z
= E , we deﬁne for ( i, j ) œ [ n ] 2 , Q Eij =
K Z ,ij /K
Z , ++ and P Dij
= P Dij + P Dji .
The symmetry of Q E yields:
[Equation]
[Equation]
and the symmetrized objective of t-SNE [38] is recovered.
LargeVis.
Now choosing P X
= D and P Z = B , one can also notice that Q B =
( K Z ,ij / (1 + K Z ,ij ))
( i,j ) œ [ n ] 2 is symmetric.
With this at hand the limit cross-entropy reads
H D,B = ≠
[Equation]
which is the objective of LargeVis [36].
[Table]
[Equation]
UMAP.
Let us take P X
= P Z = B and consider the symmetric thresholded graph Ê W X =
11 W X + W € X Ø 1 .
By independence of the edges, Ê W X ,ij ≥ B 1 Â P Bij where Â P Bij =
P Bij + P Bji ≠ P Bij P Bji and P B
= ( K X ,ij / (1 + K X ,ij ))
( i,j ) œ [ n ] 2 .
Coupling Ê W X and W Z gives:
[Equation]
which is the loss function considered in UMAP [30], the construction of Ê W X being borrowed from section 3.1 of the paper.
Remark 4 One can also consider H E,E but as detailed in [38], this criterion fails at positioning outliers and is therefore not considered.
Interestingly, any other feasible combination of the presented priors relates to an existing method.
3. Interpretations
As we have seen in section 3.1, SNE-like methods can all be derived from the graph coupling framework.
What characterizes each of them is the choice of priors considered for the latent structuring graphs.
To the best of our knowledge, the presented framework is the ﬁrst that manages to unify all these DR algorithms.
Such a framework opens many perspectives for improving upon current practices as we discuss in section 4 and section 5.
We now focus on a few insights that our work provides about the empirical performances of these methods.
Repulsion & Attraction.
Decomposing H P X , P Z with Bayes’ rule and simplifying constant terms one has the following optimization problem:
[Equation]
The ﬁrst and second terms in eq.
(5) respectively summarize the attractive and repulsive forces of the objective.
Recall from proposition 1 that P P X is the posterior expectation of W X .
Hence in SNE-like methods, the attractive forces resume to a pairwise MRF log likelihood with respect to a graph posterior expectation given X .
For instance if k z is the
Gaussian kernel, this attractive term reads tr [Equation] where [Equation]
, boiling down to the objective of Laplacian eigenmaps [5].
Therefore, for Gaussian MRFs, the attractive forces resume to an unconstrained Laplacian eigenmaps objective.
Such link, already noted in [7], is easily unveiled in our framework.
Moreover, one can notice that only this attractive term depends on X as the repulsion is given by the marginal term in (5).
The latter reads P ( Z ) =
[Equation]
P ( Z , W ) with P ( Z , W ) Ã f k
( Z , W )  P Z ( W ) .
Such penalty notably prevents a trivial solution, as 0 , like any constant vector, is a mode of f k ( · , W ) for all W .
Also note that the prior for W X only conditions attraction while the prior for W Z only a  ects repulsion.
In the present work we focus solely on deciphering the probabilistic model that accounts for neighbor embedding loss functions and refer to [6] for a quantitative study of attraction and repulsion in these methods.
Global Structure Preservation.
To gain intuition, consider that W X is observed.
As we showed in section 2.2, when one relies on shift invariant kernels, the positions of the
CC means are taken from a di  use distribution.
Since the above methods are all derived from the limit posteriors at Á æ 0 , X M and Z M have no inﬂuence on the coupling objective.
Hence if two nodes belong to di  erent CCs, their low dimensional pairwise distance will likely not be faithful.
We can expect this phenomenon to persist when the expectation on
W X is considered, especially when clusters are well distinguishable in X .
This observation is central to understand the large scale deﬁciency of these methods.
Note that this happens at the beneﬁt of the local structure which is faithfully represented in low dimension, as discussed in section 1.
In the following section we propose to mitigate the global structure deﬁciency with non-degenerate MRF models.
Towards Capturing Large-Scale Dependencies
In this section, we investigate the ability of graph coupling to faithfully represent global structure in low dimension.
To gain intuition on the case where the distribution induced by the graph is not degenerate, we consider a proper Gaussian graph coupling model and show its equivalence with PCA.
We then provide a new initialization procedure to alleviate the large scale deﬁciency of graph coupling when degenerate MRFs are used.
4. PCA as Graph Coupling
As we argue that the inability of SNE-like methods to reproduce the coarse-grain dependencies of the input in the latent space is due to the degeneracy of the conditional (3), a natural solution would be to consider graphical models that are well deﬁned and integrable on the entire deﬁnition spaces of X and Z .
For simplicity, we consider the Gaussian model and leave the extension to other kernels for future works.
Note that in this case integrability translates into the precision matrix being full-rank.
As we see with the following, the natural extension of our framework to such models leads to a well-established PCA algorithm.
In the following, for a continuous variable ⇥ Z , P ( ⇥ Z = · ) denotes its density.
Theorem 2 Let ‹ Ø n , ⇥ X ≥ W ( ‹ , I n ) and ⇥ Z ≥ W ( ‹ + p ≠ q, I n ) .
Assume that ⇥ X and ⇥ Z structure the rows of respectively X and Z such that:
[Equation]
Then the solution of the precision coupling problem:
[Equation]
is a PCA embedding of X with q components.
We now highlight the parallels with the previous construction done for neighbor embedding methods.
First note that the multivariate Gaussian with full-rank precision is inherently a pairwise MRF [35].
When choosing the Gaussian kernel for neighbor embedding methods, we saw that the graph Laplacian L X of W X was playing the role of the among-row precision matrix, as we had X | W X ≥ N ( 0 , L ≠ 1 X ¢ I p ) (equation 2).
Recall that the later always has a null-space which is spanned by the CC indicator vectors of W (section 2.1).
Here, the key di  erence is that we impose a full-rank constraint on the precision ⇥ .
Concerning the priors, we choose the ones that are conjugate to the conditionals (6) and (7), as previously done when constructing the prior for neighbor embedding methods (deﬁnition 2).
Hence in the full-rank setting, the prior simply amounts to a Wishart distribution denoted by W .
The above theorem further highlights the ﬂexibility and generality of the graph coupling framework.
Unlike usual constructions of PCA or probabilistic PCA [37], in the above the linear relation between X and Z is recovered by solving the graph coupling problem and not explicitly stated beforehand.
To the best of our knowledge, it is the ﬁrst time such a link is uncovered between PCA and SNE-like methods.
In contrast with the latter,
PCA is well-known for its ability to preserve global structure while being signiﬁcantly less e  cient at identifying clusters [2].
Therefore, as suspected in section 3.2, the degeneracy of the conditional distribution given the graph is key to determine the distance preservation properties of the embeddings.
We propose in section 4.2 to combine both graph coupling approaches to strike a balance between global and local structure preservation.
4. Hierarchical Graph Coupling
The goal of this section is to show that global structure in SNE-like embeddings can be improved by structuring the CCs’ positions.
We consider the following hierarchical model for X , where P X œ { B, D, E } , k x satisﬁes the assumptions of theorem 1 and ‹ X Ø n :
[Equation]
2 where U [ R ] are the eigenvectors associated to the Laplacian null-space of W X .
Given a graph W X , the idea is to structure the CCs’ relative positions with a full-rank Gaussian model.
The same model is considered for W Z , ⇥ Z and Z , choosing ‹ Z = ‹ X + p ≠ q for the Wishart prior to satisfy the assumption of theorem 2.
With this in place, we aim at providing a complete coupling objective, matching the pairs ( W X , ⇥ X ) and ( W Z , ⇥ Z ) .
The joint negative cross-entropy can be decomposed as follows:
[Equation]
where (8) is the usual coupling criterion of W X and W Z capturing intra-CC variability while (9) is a penalty resulting from the Gaussian structure on S M .
Constructed as such, the above objective allows a trade-of between local and global structure preservation.
Following current trends in DR [21], we propose to take care of the global structure ﬁrst i.e. focusing on (9) before (8).
The di  culty of dealing with (9) lies in the hierarchical construction of the graph and the Gaussian precision (see ﬁg. 3).
We state the following result.
Corollary 1 Let W X œ S W , L =
L ( W X ) and S q M = (ker L ) ¢ R q , then for all Á > 0 , given the above hierarchical model, the solution of the problem:
[Equation]
is a PCA embedding of U [: R ]
U €
[ R ] X where U [: R ] are the CCs’ membership vectors of W X .
Remark 5 Note that while (8) approximates the objective of SNE-like methods when Á æ 0 , the minimizer of (9) given by corollary 1 is stable for all Á .
From this observation, we propose a simple heuristic to minimize (9) that consists in computing a PCA embedding of
E P P X ( · ; K X ) #
U [: R ] U € [ R ] $ X .
The distribution of the connected components of the posterior of
W X being intractable, we resort to a MonteCarlo estimation of the above expectation.
The latter procedure called ccPCA aims at recovering the inter-CC structure that is ﬁltered by SNE-like methods.
ccPCA may then be used as initialization for optimizing (8) which is done by running the DR method corresponding to the graph priors at hand (section 3.1).
This second step essentially consists in reﬁning the intra-CC structure.
4. Experiments with ccPCA
Figure 1 shows that a t-SNE embedding of a balanced MNIST dataset of 10000 samples [14] with isotropic Gaussian initialization performs poorly in conserving the relative positions of clusters.
As each digit cluster contains approximately 1000 points, with a perplexity of 30 , sampling an edge across digit clusters in the graph posterior P P X ( · ; K X ) is very unlikely.[-PCA 1.0 0.5 0.0 0.5 1.0 1.5 2.0 2.5 2.0 1. Laplacian Eigenmaps 30 20 10 0 10 20 301010 ccPCA 60 40 20 0 20 40 
 PCA + t-SNE 60 40 20 0 20 40 6040040 
-]
Recall that the perplexity value [38] corresponds to the approximate number of e  ective neighbors of each point.
Hence images of di  erent digits are with very high probability in di  erent CCs of the graph posterior and their CC-wise means are not coupled as discussed in section 3.2.
To remedy this in practice, PCA or Laplacian eigenmaps are usually used as initialization [21].
These strategies are tested (ﬁg. 2) together with ccPCA .
This shows that ccPCA manages to retrieve the digits that mostly support the large-scale variability as measured by the peripheral positioning of digits 0 (blue), 2 (green), 6 (pink) and 7 (grey) given by the right side of ﬁg. 1.
Other perplexity values for ccPCA are explored in appendix B.2 while the experimental setup is detailed in appendix B.1.
In appendix B.3, we perform quantitative evaluations of ccPCA for both t-SNE and UMAP on various datasets using K-ary neighborhood criteria.
We ﬁnd that using ccPCA as initialization is in general more reliable than PCA and Laplacian eigenmaps for preserving global structure using both t-SNE and UMAP.
Compared to PCA, ccPCA manages to aggregate points into clusters, thus ﬁltering the intra-cluster variablity and focusing solely on the inter-cluster structure.
Compared to
Laplacian eigenmaps which performs well at identifying clusters but su  ers from the same deﬁciency as t-SNE for positioning them, ccPCA retains more of the coarse-grain structure.
These observations support our unifying probabilistic framework and the theoretical results about the MRF degeneracy which are the leading contributions of this article.
The ccPCA initialization appears as a ﬁrst stepping stone towards more grounded DR methods based on the probabilistic model presented in this article.
Conclusion and Perspectives
In this work we shed a new light on most popular DR methods by showing that they can be uniﬁed within a common probabilistic model in the form of latent Markov Random Fields
Graphs coupled by a cross entropy.
The deﬁnition of such a model constitutes a major step towards the understanding of common dimension reduction methods, in particular their structure preservation properties as discussed in this article.
Our work o  ers many perspectives, among which the possibility to enrich the probabilistic model with more suited graph priors.
Currently considered priors are simply the ones that are conjugate to the MRFs thus they are mostly designed to yield a tractable coupling objective.
However they may not be optimal and could be modiﬁed to capture targeted features, e.g. communities, in the input data, and give adapted representations in the latent space.
The graph coupling approach could also be extended to more general latent structures governing the joint distribution of observations.
Finally, the probabilistic model could be leveraged to tackle hyper-parameter calibration, especially kernel bandwidths that have a great inﬂuence on the quality of the representations and are currently tuned using heuristics with unclear motivations.
Acknowledgments.
The authors would like to thank the anonymous reviewers whose comments and questions helped improve the clarity of this manuscript, as well as Aurélien
Garivier, Antoine Barrier and Floshi Poshi for helpful discussions.
This work was supported by the Agence Nationale de la Recherche ANR-18-CE45-0023 SingleStatOmics
[-( X i − X j ) 
-]
[-where Q n ( K ) = 1 Kn P ni =
-]