E FFECTIVE S UBSPACE I NDEXING VIA I NTERPOLATION ON S TIEFEL AND G RASSMANN MANIFOLDS
Anonymous authors
Paper under double-blind review
A BSTRACT
We propose a novel local Subspace Indexing Model with Interpolation ( SIM-I ) for low-dimensional embedding of image data sets.
Our SIM-I is constructed via two steps: in the ﬁrst step we build a piece-wise linear afﬁnity-aware subspace model under a given partition of the data set; in the second step we interpolate between several adjacent linear subspace models constructed previously using the “center of mass” calculation on Stiefel and Grassmann manifolds.
The resulting subspace indexing model built by SIM-I is a globally non-linear low-dimensional embedding of the original data set.
Furthermore, the interpolation step produces a “smoothed” version of the piece-wise linear embedding mapping constructed in the ﬁrst step, and can be viewed as a regularization procedure.
We provide experimental results validating the effectiveness of SIM-I , that improves PCA recovery for SIFT data set and nearest-neighbor classiﬁcation success rates for MNIST and CIFAR-10 data sets.
I NTRODUCTION
Subspace selection algorithms have been successful in many application problems related to dimension reduction (Zhou et al.
(2010), Bian & Tao (2011), Si et al.
(2010), Zhang et al.
(2009)), with applications including, e.g., human face recognition (Fu & Huang (2008)), speech and gait recognition (Tao et al.
(2007)), etc..
The classical approaches of subspace selection in dimension reduction include algorithms like Principle Component Analysis (PCA, see Jolliffe (2002)) and Linear Discriminant Analysis (LDA, see Belhumeur et al.
(1997), Tao et al.
They are looking for globally linear subspace models.
Therefore, they fail to estimate the nonlinearity of the intrinsic data manifold, and ignore the local variation of the data (Saul & Roweis (2003), Strassen (1969)).
Consequently, these globally linear models are often ineffective for search problems on large scale image data sets.
To resolve this difﬁculty, nonlinear algorithms such as kernel algorithms (Ham et al.
and manifold learning algorithms (Belkin et al.
(2006), Guan et al.
(2011)) are proposed.
However, even though these nonlinear methods signiﬁcantly improve the recognition performance, they face a serious computational challenge dealing with large-scale data sets due to the complexity of matrix decomposition at the size of the number of training samples.
Here we propose a simple method, Subspace Indexing Model with Interpolation ( SIM-I ), that produces from a given data set a piece-wise linear, locality-aware and globally nonlinear model of low-dimensional embedding.
SIM-I is constructed via two steps: in the ﬁrst step we build a piecewise linear afﬁnity-aware subspace model under a given partition of the data set; in the second step we interpolate between several adjacent linear subspace models constructed previously using the “center of mass” calculation on Stiefel and Grassmann manifolds (Edleman et al.
, Kaneko et al.
, Marrinan et al.
The interpolation step outputs a “smoothed” version (Figure 1) of the original piece-wise linear model, and can be regarded as a regularization process.
Compared to previously mentioned subspace methods, SIM-I enjoys the following advantages: (1) it captures the global nonlinearity and thus the local ﬂuctuations of the data set; (2) it is computationally feasible to large-scale data sets since it avoids the complexity in matrix decomposition at the size of the number of training samples; (3) it includes a regularization step via interpolating between several adjacent pieces of subspace models.
Numerical experiments on PCA recovery task for SIFT data set and classiﬁcation tasks via nearest-neighbor method for MNIST and CIFAR-10 data sets further validate the effectiveness of SIM-I .
P IECE - WISE LINEAR L OCALITY P RESERVING P ROJECTION (LPP) MODEL
If an image data point x ∈ R D is represented as a vector in a very high-dimensional space, then we want to ﬁnd a low-dimensional embedding y = f ( x ) ∈ R d , d << D such that the embedding function f retains some meaningful properties of the original image data set, ideally close to its intrinsic dimension.
If we restrict ourselves to linear maps of the form y = W T x ∈ R d , where the
D × d projection matrix W = ( w ij ) 1 ≤ i ≤ D, 1 ≤ j ≤ d (assuming full rank), then such a procedure is called a locally linear low-dimensional embedding (see Roweis & Saulm (2000); Van Der Maaten et al.
The target is to search for a “good” projection matrix W , such that the projection x (cid:55)→
y
= W T x must preserve certain locality in the data set (this is called a Locality Preserving Projection , or LPP projection, see He & Niyogi (2003)).
The locality is interpreted as a kind of intrinsic relative geometric relations between the data points in the original high-dimensional space, usually represented by the afﬁnity matrix S
=
( s ij ) 1 ≤
i,j ≤ n (which is a symmetric matrix with non-negative terms).
As an example, given unlabelled data points x 1 , ..., x n ∈ R D , we can take
[Equation] (cid:17) when (cid:107) x
i − x j (cid:107) <
ε and s ij =
otherwise.
Here σ > 0 and ε > 0 is a small threshold parameter, and (cid:107) x
i − x j (cid:107) is the Euclidean norm in R D .
Based on the afﬁnity matrix S =
( s ij ) , the search for the projection matrix W can be formulated as the following optimization problem
[Equation]
in which y i =
W T x i and y j = W T x j and the norm (cid:107)
y i − y j (cid:107) is taken in the projected space R d .
Usually when (cid:107) x i − x j (cid:107) is large, the afﬁnity s ij will be small, and vice versa.
Thus (1) is seeking for the embedding matrix W such that close pairs of image points x i and x j will be mapped to close pairs of embeddings
y i
= W T x i and y j = W T x j , and vice versa.
This helps to preserve the local geometry of the data set, a.k.a the locality.
To solve (1), we introduce a weighted fully-connected graph G where the vertex set consists of all data points x 1 , ..., x n and the weight on the edge connecting x i and x j is given by s ij ≥ 0 .
Consider the diagonal matrix D = diag ( D 11 , ..., D nn ) where
D ii = n (cid:80) j
=1 s ij , and we then introduce the graph Laplacian L = D − S .
Then the minimization problem (1), together with the normalization constraint n (cid:80) i [Equation] i = 1 , reduces to the following generalized eigenvalue problem (see He & Niyogi (2003))
[Equation]
where X =
[ x 1 , ..., x n ] ∈ R D × n .
Assume we have obtained an increasing family of eigenvalues 0 = λ 0 < λ 1 ≤ ... ≤ λ n − 1 .
Let the corresponding eigenvectors be w 0 , w 1 , ..., w n − 1 .
Then the low-dimensional embedding matrix can be taken as W =
[ w 1 , ..., w d ] (see He et al.
By choosing different afﬁnity matrices S =
( s ij ) , the above LPP framework includes many commonly seen practical examples.
For example, if the data x 1 , ..., x n are not labelled, then we can take s ij = 1 n and (2) produces the classical Principle
Component Analysis (PCA).
For labelled data forming subsets X 1 , ..., X m with same labels in each subset, we can take s ij = 1 n k when x i , x j ∈ X k , and s ij = 0 other-wise.
Here n k is the cardinality of X k .
This will produce Linear Discriminant Analysis (LDA).
The detailed justiﬁcations of these connections can be found in He et al.
Given an input data set X = { x 1 , ..., x n } where each x i ∈ R D , either labelled or unlabelled, we can apply a k-d tree (Bentley (1975), Wang et al.
based partition scheme to divide the whole data set X into non-overlapping subsets C 1 , ..., C 2 h where h is the depth of the tree.
Conventional subspace selection algorithms could be applied on the whole sample space before the whole space is partitioned and indexed.
For example, we can ﬁrst apply a PCA to X , which selects the ﬁrst d bases [ a 1 , ..., a d ] with largest variance.
Based on these bases, the covariance information obtained from global PCA is utilized in the indexing as follows: (1) we project all sample points x 1 , ..., x n onto the maximum variance basis a 1 , then we ﬁnd the median value m 1 of the projected samples, and split the whole collection of data along a 1 at m 1 , i.e., split the current node into left and right children; (2) starting from level i = 2 , for each left and right child, project the whole collection of data along the i -th maximum variance basis a i , ﬁnd the median value m i , and split all the children at m i ; (3) increase the level from i to i + 1 and repeat (2) until i =
h reaches the bottom of the tree.
We collect all the 2 h children at level i = h and obtain the disjoint subsets C 1 , ..., C 2 h .
Each subset
C k , k = 1 , 2 , ..., 2 h consists of a family of input data in R D .
Based on them, using the above LPP framework, for each C k , a low-dimensional embedding matrix W k ∈ R D × d can be constructed.
In this way, over the whole data set X , we have constructed a piece-wise linear low-dimensional embedding model f ( x ) : R D
→ R d (see Figure 1(a)) where x ∈ X .
This model is given by the linear embedding matrices W 1 , ..., W 2 h ∈ R D × d .
The above model construction can be regarded as a training process from the data set X .
For a given test data point x ∈ R D , not included in X , we can ﬁnd the closest subset C k ( x ) to it, by selecting the index k
= k ( x ) ∈ { 1 , ..., 2 h } with the smallest distance (cid:107) x
− m k
( x ) (cid:107) .
Here m k is the mean among all data points in the subset C k .
With the subset C k ( x ) chosen, we map the test point x ∈ R D to its low-dimensional embedding f ( x ) =
W Tk ( x ) x ∈ R d .
Such a procedure extended the piece-wise linear embedding model f ( x ) : R D → R d to all testing data points in R D .
C ALCULATING THE “ CENTER OF MASS ” ON S TIEFEL AND G RASSMANN MANIFOLDS
Subspace Indexing (Wang et al.
(2011)) provides a d -dimensional representation of the data set { x 1 , ..., x n } by the subspace span ( w 1 , ..., w d )
= { W T x, x ∈ R D } generated from the linear embedding matrix W ∈ R D × d .
In this case, we are only interested in the column space of W , so we can assume that w 0 , w 1 , ..., w n − 1 is an orthonormal basis 1 .
Such a matrix W belongs to the Stiefel manifold , deﬁned by
Deﬁnition 1 (Stiefel manifold)
The compact Stiefel manifold St( d, D ) is a submanifold of the Eucilidean space R D ×
d such that
[Equation]
As an example, if we are interested in signal recovery using low-dimensional PCA embedding, the projections we calculated from PCA analysis will be on Stiefel manifolds.
However, for classiﬁcation tasks, the exact distance information is less important than label information.
In this case, two such Stiefel matrices W 1 and W 2 produce the same embedding if W 1 =
W 2 O d for some O d ∈ O ( d ) , where O ( d ) is the group of orthogonal matrices in dimension d .
In this case, the relevant embedding we obtained is a point on the Grassmann manifold , deﬁned by
Deﬁnition 2 (Grassmann manifold)
The Grassmann manifold Gr( d, D ) is deﬁned to be the quotient manifold Gr( d, D ) =
St( d, D ) /O
( d ) .
A point on Gr( d, D ) is deﬁned by an equivalence class [ W ] = { WO d , O d ∈ O ( d ) } where W ∈ St( d, D ) .
Given a family of elements on the Stiefel or Grassmann manifold, the center-of-mass is deﬁned as an element on the same manifold that minimizes the functional given by the weighted sum of square distances.
To be precise, we have
Deﬁnition 3 (Stiefel and Grassmann center-of-masses)
Given a sequence of matrices W 1 , ..., W l ∈ St( d, D ) and a sequence of weights w 1 , ..., w l > 0 , the Stiefel center-of-mass with respect to the distance d
W 1 , W 2 ) on St( d, D ) is deﬁned as a matrix W c
=
W St c ( W 1 , ..., W l ; w 1 , ..., w l ) ∈
St( d, D )
such that
[Equation]
Similarly, if the corresponding equivalent classes are [ W 1 ] , ..., [
W l ] ∈ Gr( d, D ) , then the Grassmann center-of-mass with respect to the distance d([ W 1 ] , [ W 2 ]) on Gr( d, D ) is deﬁned as the equivalence class [ W c ] , where W c = W Gr c ( W 1 , ..., W l ; w 1 , ..., w l ) ∈
St( d, D ) is such that
[Equation]
The distances d( W 1 , W 2 ) or d([ W 1 ] , [ W 2 ]) can be taken in different ways.
For example, for
W 1 , W 2 ∈ St( d, D ) , one way is to consider d( W 1 , W 2 ) =
d F ( W 1 , W 2 )
= (cid:107) W 1 − W 2 (cid:107) F , the matrix Frobenius norm of W 1 − W 2 .
One can also take a more intrinsic distance, such as the geodesic distance between W 1 and W 2 on the manifold St( d, D ) with the metric given by embedded geometry (see Edleman et al.
For [ W 1 ] , [ W 2 ] ∈ Gr( d, D ) , one way is to consider the projected
Frobenius norm d([ W 1 ] , [ W 2 ]) =
d pF ([ W 1 ] , [ W 2 ]) =
2 − 1 / 2 (cid:107) W 1 W T 1 − W 2 W T 2 (cid:107) F .
There are also many other choices, such as using the principle angles between the subspaces, chordal norms, or other types of Frobenius norms (see Edleman et al.
(1999, Section 4.3)).
With respect to matrix Frobenius norm and projected Frobenius norm, the Stiefel and Grassmann center-of-masses can be calculated explicitly in the following theorems.
Theorem 1 (Stiefel center-of-mass with respect to Frobenius norm)
We consider the singular value decomposition of the matrix (cid:80) lj
=
1 w j W j
=
O
1 ∆ O 2 , where O 1 ∈ O ( D ) and O 2 ∈ O
( d ) , ∆ = (cid:18) diag ( λ 1 , ...,
λ d )
d × d 0 ( D − d ) × d (cid:19) and λ 1 ≥ ... ≥ λ d ≥ 0 are the singular values.
Then the Stiefel center-of-mass with respect to the distance given by Frobenius norm d( W 1 , W 2 ) =
(cid:107) W 1 − W 2 (cid:107) F is given by W c
= O 1 Λ
O 2 where Λ = (cid:18) diag (1 , ..., 1)
d × d 0 ( D − d ) × d (cid:19) .
Theorem 2 (Grassmann center-of-mass with respect to projected Frobenius norm) Set Ω
j = (cid:16)(cid:80) lj
=1 w j (cid:17) − 1 w j .
We consider the singular value decomposition of the symmetric matrix l (cid:80) j =1 Ω j W j W Tj = Q ∆ Q T where Q ∈ O
( D ) and ∆ =
diag ( σ 21 , ..., σ 2 D ) , σ 21 ≥ ... ≥ σ 2 D ≥ 0 .
Then the Grassmann center-of-mass with respect to the distance given by projected Frobenius norm d pF ([ W 1 ] , [ W 2 ]) =
2 − 1 / 2 (cid:107) W 1 W T 1 − W 2 W T 2 (cid:107) F is the equivalence class [ W c ] determined by W c = Q Λ , where Λ = (cid:18) diag (1 , ..., 1)
d × d 0 ( D − d ) × d (cid:19) .
SIM-I : I NTERPOLATING THE LPP MODEL FAMILY
Recall that we have developed a piece-wise linear embedding model f ( x ) : R D → R d over the data set X = { x 1 , ..., x n } .
The embedding f ( x ) corresponds to a family of subspace indexing models
W 1 , ..., W 2 h ∈ St( d, D ) (e.g. for PCA signal recovery tasks) or [ W 1 ] , ..., [ W 2 h ]
∈
Gr( d, D ) (e.g. for classiﬁcation tasks).
Each subspace model W k is built from LPP embedding using the subset
C k ⊂ X developed from k-d tree and h is the depth tree.
Given a test point x ∈ R D that does not lie in X , we can map it to the low-dimensional embedding f ( x ) =
W Tk ( x ) x ∈ R d .
The index k ( x ) corresponds to the subset C k ( x ) that lie closest to x .
In practice, we can ﬁrst compute the means m k over all the data points in the subset C k for each k
= 1 , 2 , ..., 2 h and sort the distances (cid:107) x − m k (cid:107) in ascending order (cid:107) x − m k 1 ( x ) (cid:107) ≤ ... ≤ (cid:107) x − m k 2 h
( x ) (cid:107) , { k 1 ( x ) , ..., k 2 h ( x ) } = { 1 , ..., 2 h } .
We then take k ( x ) =
k 1 ( x ) to be the index k corresponding to the shortest distance.
This is effective when the test point x lies signiﬁcantly close to one of the subsets C k
( x ) , see Figure 2(a).
k i ( x
i
the embedding matrices W j 1 ( x ) , ..., W j I ( x )
St( d, D ) or their corresponding subspaces [ W j 1 ( x ) ] , ..., [ W j I ( x ) ]
∈ Gr( d, D ) , together with the weights w i = exp( − K (cid:107) x − m j i ( x ) (cid:107) 2 ) > 0 for i =
However, for a general test point x ∈ R D , it might happen that this point lies at approximately the same distances to the centers of each of the several different subsets adjacent to x (see Figure 2(b)).
In this case, we aim to interpolate between several subspace indexing models W j 1 ( x ) , ..., W j I ( x ) .
To do this, we ﬁrst ﬁnd the subspace indexes j 1 ( x ) , ..., j I ( x ) from the ﬁrst I subsets C j 1 ( x ) , ..., C j I ( x ) closest to x , i.e., j 1 ( x ) =
k 1 ( x ) , ..., j I ( x ) =
k I ( x ) given the sorted distances (cid:107) x − m k (cid:107) mentioned above.
In practice, the number I
= I ( x ) is depending on x and can be chosen in the following way:
I is the ﬁrst sub-index i of k
i ( x ) such that (cid:107) x − m k
I +1 ( x ) (cid:107) >
r thr (cid:107) x − m k 1 ( x ) (cid:107) , where r thr > 1 is a threshold ratio that can be tuned.
We then pick the weights as w i = exp( − K (cid:107) x − m j i ( x ) (cid:107) 2 ) for some
K >
and i = 1 , 2 , ..., I .
This is indicating that the closer x is to C j i ( x ) , the heavier weights we assign to W j i ( x ) in the interpolation process.
Given the embedding matrices W j 1 ( x ) , ..., W j I ( x ) ∈
St( d, D ) or their corresponding subspaces [ W j 1 ( x ) ] , ..., [ W j I ( x ) ]
∈ Gr( d, D ) , together with the weights w 1 , ..., w I > 0 , we ﬁnd a center-of-mass W c =
W St c ( W j 1 ( x ) , ..., W j I ( x ) ; w 1 , ..., w I ) (Stiefel case) or [ W c ] =
[ W Gr c ( W j 1 ( x ) , ..., W j I ( x ) ; w 1 , ..., w I )]
(Grassmann case) according to Definition 3 and Theorems 1 and 2.
Finally, we map the test point x to the low-dimensional embedding f ( x ) =
W Tc x ∈ R d .
Notice that when I = 1 , the interpolation procedure reduces to projecting x using W k 1 ( x ) calculated from LPP analysis on the closest subset only.
In general, the whole interpolation procedure can be regarded as providing a regularized version of the piece-wise linear embedding we discussed in Section 2 (also see Figure 1).
We summarize our interpolation method as the SIM-I Algorithm 1.
E XPERIMENTS 5. PCA R ECOVERY FOR SIFT DATA SET
SIFT (Scale Invariant Feature Transform, see Lowe (2004)) data set is a data set that computes for each keypoint a real-valued descriptor, based on the content of the surrounding patch in terms of local intensity gradients.
Given its remarkable performance, SIFT has been often used as starting point for the creation of other descriptors.
The ﬁnal SIFT descriptor consists of 128 elements.
This means that each data point in SIFT data set has dimension D = 128 , and we pick the embedding dimension d = 16 .
The original SIFT data set has 10068850 data samples, that form the data set sift sample .
We randomly collect n train = 200 × 2 13 elements from these sample points as our target data set X = sift train = { x 1 , ..., x n train ∈ R D } .
We consider the recovery efﬁciency of PCA embedding of the SIFT data set.
Let x be a point in sift sample and let
W be a Stiefel matrix in St(16 , 128) .
The projection of x onto the 16 -dimensional space is then denoted by y = W T x .
By recovery we meant to consider the point (cid:98) x
= ( W T )
− y where ( W T ) − is the Moore-Penrose pseudo-inverse of W T .
The recovery efﬁciency can then be quantiﬁed by the recovery error (cid:107) x − (cid:98) x (cid:107) , where the Euclidean norm is computed in R 128 .
We pick h = 13 so that 2 h = 8192 .
Then we decompose sift train into 8192 subsets C 1 , ..., C 8192 using k-d tree based partition.
For each subset C k , we calculate the mean m k ∈ R
and we obtain a PCA embedding matrix W k ∈ St(16 , 128) .
We sort the distances (cid:107) x − m k
(cid:107) , k
= 1 , 2 , ..., 8192 in ascending order so that (cid:107) x − m k 1 (cid:107) ≤ ... ≤ (cid:107) x − m k 8192 (cid:107) , where { k 1 , ..., k 8192 } = { 1 , ..., 8192 } .
Then we ﬁnd among the subset means m k , k = 1 , 2 , ..., 8192 the ﬁrst I nearest to x , with their indexes denoted by k i
=
k
i ( x ) , i = 1 , .., I .
The number I
=
I ( x ) is depending on x and is chosen in the following way: I is the ﬁrst sub-index i of k i such that [Equation]
> r thr (cid:107) x
[Equation] m k 1 (cid:107) , where r thr > 1 is a threshold ratio.
We pick r thr = 2 .
For test data set, we randomly pick from sift sample \ sift train a subset of size n test = 500 , and we denote the data set as sift test .
For each test point x ∈ sift test , we consider sending it to the nearest subset C k 1 and the corresponding Stiefel matrix is W k 1 ∈ St(16 , 128) .
We can then consider the recovery point (cid:98) x
= ( W Tk 1 ) − W Tk 1 x and the benchmark recovery error
Error bm = (cid:107) x − (cid:98) x (cid:107) .
Consider the alternative recovery scheme using our method SIM-I .
We calculate the weights w j = exp( − K (cid:107) x
− m k j (cid:107) 2 ) for j = 1 , ..., I and we choose the constant K = 10 − 8 .
We then ﬁnd
the
Stiefel center-of-mass W c =
W St c ( W k 1 ( x ) , ..., W k I ( x ) ; w 1 , ..., w I ) using Theorem 1 and taking the distance function to be the matrix
Frobenius norm d( W 1 , W 2 ) =
(cid:107) W 1 − W 2 (cid:107) F .
We consider the recovery point (cid:98) x
c = (( W c ) T ) − ( W c ) T x and the recovery error Error c
= (cid:107) x − (cid:98) x
c (cid:107) .
Over the test set sift test , we ﬁnd that for about 94 .
2% test points, Error c < Error bm, which implies that SIM-I improved the efﬁciency of recovery.
The empirical average Error c is [Figure] [Figure] 089506 , and the empirical average Error bm is 454 .
Figure 3 plots the Error c and
Error bm (vertical axis) as functions of the test sample indexes.
The red curve is for Error c and blue curve is for Error bm, where we have sorted Error bm in ascending order and reordered the test indexes correspondingly.
Figure 4 gives the differences Error c − Error bm in descending order.
It can be apparently seen that Error c − Error bm < 0 for most of test samples.
5. N EAREST -N EIGHBOR C LASSIFICATION FOR MNIST AND CIFAR-
Here we consider a labeled data set data = { ( x i , y i ) , i = 1 , 2 , ..., N } where x i ∈ R D are the inputs and y i ∈ N are the labels.
We randomly select the training set data train = { ( x i , Y i ) , i = 1 , 2 , ..., n train } and the test set data test =
{ ( a i , b i ) , i = 1 , 2 , ..., n test } , and we make them disjoint
We ﬁrst project the training set onto a kd PCA -dimensional subspace via a standard PCA.
Based on this initial embedding, using a kd-tree with height h , we divide data train into 2 h clusters C k , k = 1 , 2 , ..., 2 h .
For each C k , we ﬁnd an LPP embedding matrix W k ∈ St( kd LPP , D ) by setting the afﬁnity matrix to be s ij = exp( −(cid:107) x
i − x j (cid:107) 2 ) if x i , x j ∈ C k are in the same class and s ij = 0 otherwise.
Since we are having a classiﬁcation problem, we can identify each W k by the subspace it spans, i.e., we consider [ W k ] which is the equivalence class of W k in Gr( kd LPP , D ) .
As before, we compute in R D the means m k of each cluster C k .
For a test point x ∈ data test , we sort the distances (cid:107) x − m k
(cid:107) , k
= 1 , 2 , ..., 2 h in ascending order so that (cid:107) x − m k 1 (cid:107) ≤ ... ≤ (cid:107) x − m
k 2 h (cid:107) , where { k 1 , ..., k 2 h } = { 1 , ..., 2 h } .
Then we ﬁnd among the cluster means m k , k
= 1 , 2 , ..., 2 h the ﬁrst I nearest to x , with their indexes denoted by k i =
k i ( x ) , i = 1 , ..., I .
The number I
=
I ( x ) is depending on x and is chosen in the following way: I is the ﬁrst sub-index i of k i such that (cid:107) x − m k
I +1 (cid:107) >
r thr (cid:107) x − m k 1 (cid:107) , where r thr > 1 is a threshold ratio.
The parameter r thr can be treated as a hyper-parameter that we can tune here.
For baseline method, we do a nearest-neighbor classiﬁcation for x on a low-dimensional embedding of the cluster C k 1 , and we pick the number of nearest-neighbors to be knn ≥ 1 .
Indeed we project x and all training data points in C k 1 using W k 1 , and perform nearest-neighbor classiﬁcation on the resulting projection.
For our method SIM-I , we take the union C k 1 ∪ ... ∪ C k
I .
Recall that each C k i corresponds to an LPP embedding projection matrix W k i .
We set the weights w i = exp( − K (cid:107) x − m k
i (cid:107) 2 ) , and we pick K = 10 − 8 .
We compute a center-of-mass W c of the projection matrices W k i with weights w i , i = 1 , 2 , ..., I using the Grassmann center-of-mass method, where the distance is taken as the projected Frobenius norm of Grassmann matrices, i.e., d( W 1 , W 2 ) =
2 − 1 / 2 (cid:107) W 1 W T 1 − W 2 W T 2 (cid:107) F and (cid:107) • (cid:107) F is the matrix Frobenius norm.
We obtain W c = W Gr c
( W k 1 ( x ) , ..., W k I ( x ) ; w 1 , ..., w I )
and we project x and all training data points in the union C k 1 ∪ ... ∪ C k
I using W c .
We then perform a nearest-neighbor classiﬁcation for x on this low-dimensional embedding with the number of nearest-neighbors being equal to knn .
[Table]
We have been experimenting on 2 different data sets, for both of them we pick kd PCA = 128 , kd LPP = 100 : (1) The MNIST data set, with N = 70000 , n train = 60000 , n test = 10000 data points and D = 784 ; (2) The CIFAR-10 data set, with N = 60000 , n train = 50000 , n test =data points and D = 3072 .
Table 1 shows the results, where the last 2 columns are the nearestneighbor classiﬁcation success rates for baseline method, and for SIM-I using Grassmann centerof-mass and the rows are for different experiments.
The ﬁrst 4 columns are for data sets, kd-tree height, threshold value r thr , the number of nearest-neighbors knn , respectively.
We slightly tuned r thr to reach best performances.
Clearly, SIM-I has its advantage.
A P ROOF OF T HEOREM
Let D ≥ d ≥ 1 .
Recall St( d, D ) stands for the Stiefel manifold, that is, each matrix in St( d, D ) is a D by d matrix with columns being orthogonally normal.
For any matrix M , recall that (cid:107) M (cid:107) 2 F =
[ tr ( M T M )] 1 / 2 is the Frobenius norm of M .
Theorem 1 (Stiefel center-of-mass with respect to Frobenius norm)
We consider the singular value decomposition of the matrix l (cid:80) j =1 w j W j
= O 1 ∆ O 2 , where O 1 ∈ O ( D ) and O 2 ∈ O
( d ) , ∆ = (cid:18) diag ( λ 1 , ...,
λ d )
d × d 0 ( D − d ) × d (cid:19) and λ 1 ≥ ... ≥ λ d ≥ 0 are the singular values.
Then the Stiefel center-of-mass with respect to the distance given by Frobenius norm d( W 1 , W 2 ) =
(cid:107) W 1 − W 2 (cid:107) F is given by W c
= O 1 Λ
O 2 where Λ = (cid:18) diag (1 , ..., 1)
d × d 0 ( D − d ) × d (cid:19) .
Proof.
For W ∈ St( d, D ) we deﬁne f ( W ) =
l [Equation]
1 w j [Equation] W − W j (cid:107) 2 F , and we are looking for a minimizer W c of f on St( d, D ) .
Write
[Equation]
It follows that
[Equation]
where B := l (cid:80) j =1 w j W j .
By singular value decomposition, there are D × D orthogonal matrix Oand d × d orthogonal matrix
O 2 such that B
= O 1 ∆
O 2 , where ∆ := (cid:18) diag ( λ 1 , ..., λ d ) d × d 0 ( D − d ) × d (cid:19) , and λ 1 ≥ · · · ≥ λ d ≥ 0 are the singular values of B .
It follows that tr ( W T B ) = tr (∆ O 2 W T O 1 ) .
Observe that ( O 2 W T O 1 ) T ∈ St( d, D ) .
Write O 2 W T O 1 = ( c ij ) d
× D .
Then D (cid:80) j
=1 c 2 ij = 1 for each i .
Hence [Equation]
(cid:88)
[Equation]
(cid:88)
[Equation]
with equality holds when O 2 W T O
= ( I d , O ) where O is the d × ( D − d ) matrix with all entries being zero.
This says that W c
=
O 1 Λ O 2 where Λ = (cid:18) diag (1 , ..., 1)
d × d 0 ( D − d ) × d (cid:19) is the maximizer of tr ( W T B ) .
The so obtained W c serves as the minimizer of f ( W ) on St( d, D ) and is thus the centerof-mass.
(cid:3)
B P ROOF OF T HEOREM
Recall that Gr( d, D ) =
St( d, D ) /O
( d ) .
Every point W on St( d, D ) will correspond to an equivalence class [ W ] = { WO d : O d ∈ O ( d ) } which is a point on Gr( d, D ) .
To represent points on
Gr( d, D ) using matrices, we notice that every point on Gr( d, D ) corresponds to a unique choice of matrix WW T where W ∈ St( d, D ) .
In this way, we can deﬁne the projected Frobenius distance between two classes [ W 1 ] and [ W 2 ] in Gr( d, D ) as d 2 pF ([ W 1 ] , [ W 2 ]) =
2 − 1 / 2 (cid:107) W 1 W T 1 − W 2 W T 2 (cid:107) F .
Theorem 2 (Grassmann center-of-mass with respect to projected Frobenius norm) Set Ω j = (cid:16)(cid:80) lj
=1 w j (cid:17) − 1 w j .
We consider the singular value decomposition of the symmetric matrix l (cid:80) j =1 Ω j W j W Tj = Q ∆ Q T where Q ∈ O
( D ) and ∆ =
diag ( σ 21 , ..., σ 2 D ) , σ 21 ≥ ... ≥ σ 2 D ≥ 0 .
Then the Grassmann center-of-mass with respect to the distance given by projected Frobenius norm d pF ([ W 1 ] , [ W 2 ]) =
2 − 1 / 2 (cid:107) W 1 W T 1 − W 2 W T 2 (cid:107) F is the equivalence class [ W c ] determined by W c = Q Λ , where Λ = (cid:18) diag (1 , ..., 1)
d × d 0 ( D − d ) × d (cid:19) .
Proof.
Set M = WW T and M j = W j W Tj , then we are looking for
a minimizer W c ∈ St( d, D ) of f ( W ) ≡ (cid:80) lj
=
1 w j d 2 pF ([ W ] , [ W j ]) =
2 − 1 l (cid:80) j
=1 w j (cid:107) M − M j (cid:107) 2 F .
For two D × D matrices M 1 , Mwe deﬁne (cid:104) M 1 , M 2 (cid:105) =
tr ( M T 1 M 2 ) .
It is easy to verify that (cid:104) , (cid:105) is an inner product and (cid:107) M (cid:107) 2 F = (cid:104) M, M (cid:105) .
Thus we write 2 f ( W ) = l (cid:80) j =1 w j (cid:107) M − M j (cid:107) 2 F = l (cid:80) j =1 w j (cid:104) M, M (cid:105)
[Equation]
M j [Equation] 2 F . So the minimum is taken when (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)
[Equation]
j =1 Ω [Equation] M j [Equation]
F is minimized.
Since each W j W Tj is a symmetric matrix, we consider the SVD decomposition l (cid:80) [Equation] Ω [Equation]
W [Equation] W Tj = Q ∆ Q T where Q ∈ O ( D ) and ∆ =
diag ( σ 21 , ..., σ 2 D ) , σ 21 ≥ σ 22 ≥ ... ≥ σ 2 D ≥ 0 .
Thus it sufﬁces to set W c = arg
[Equation]
St
[Equation]
[Equation]
(cid:107)
[Equation]
−
Q ∆ Q T (cid:107) 2 F .
Since W ∈ St( d, D ) , we have WW T = PV P T where P is an orthogonal matrix of size D × D and V = diag (1 , 1 , .., 1 , 0 , 0 , ..., 0) is an D × D matrix with rank ( V ) =
d .
Moreover,
P can be chosen as P =
( W Z ) where Z is an D × ( D − d ) matrix.
So
[Equation]
Let the orthgonal matrix O =
Q − 1 P .
Then we further have
[Equation]
We then show that min O ∈ O ( n ) tr ( OV O T − ∆) 2 is achieved at the orthogonal matrix
[Equation]
for any O p × p ∈
O ( p ) and any O ( n − p ) × ( n − p ) ∈
O ( n − p ) .
Indeed, we have tr ( OV O T − ∆) 2 = tr ( OV O T ) + tr (∆ 2 ) − 2 tr (∆ OV O T ) , so that we only have to maximize tr (∆ OV O T ) =
D (cid:80) i =1 σ 2 i c ii where OV O T =
( c ij ) D
× D .
Let O =
( v ij ) D × D , then it is easy to calculate that c ii = d (cid:80)
j
=1 v 2 ij .
Moreover since D (cid:80) j =1 v 2 ij = 1 for all i and
D (cid:80) i =1 v 2 ij = 1 for all j , we know that 0 ≤ c ii ≤ 1 and
D (cid:80) i =1 c ii = d .
Thus from σ 21 ≥ ... ≥ σ 2 D ≥
we see tr (∆ OV O T ) =
D (cid:80) i =1 σ 2 i c ii ≤ d (cid:80) i =1 σ 2 i with equality if and only if c ii = 1 for i = 1 , ..., d and c ii = 0 for i =
d + 1 , ..., D .
This gives that, for
O = ( v ij ) D × D , we have d (cid:80) j =1 v 2 ij = 1 when i = 1 , ..., d and
D (cid:80) j
= d +1 v 2 ij = 1 when i =
d + 1 , ..., D .
Thus the minimum of min O ∈ O ( n ) tr ( OV O T − ∆) 2 is achieved at O ∗ = (cid:18) O d × d
O ( D − d ) ×
( D − d ) (cid:19) for any O d × d ∈ O ( d ) and any O ( D − d ) ×
( D − d ) ∈
O
( D − d ) .
We can pick O d
×
d = diag (1 , ..., 1)
d × d , and since P ∗ =
QO ∗ =
( W c Z ) where Z is an D × ( D − d ) matrix, we get the conclusion of the
Theorem about W c .
(cid:3)