L OW - RANK M ATRIX R ECOVERY WITH U NKNOWN
C ORRESPONDENCE
Anonymous authors
Paper under double-blind review
A BSTRACT
We study a matrix recovery problem with unknown correspondence: given the observation matrix M o =
[ A; ~ PB ] , where ~ P is an unknown permutation matrix, we aim to recover the underlying matrix M =
[ A; B ] .
Such problem commonly arises in many applications where heterogeneous data are utilized and the correspondence among them are unknown, e g., due to privacy concerns.
We show that it is possible to recover M via solving a nuclear norm minimization problem under a proper low-rank condition on M , with provable non-asymptotic error bound for the recovery of M .
We propose an algorithm, M 3 O (Matrix recovery via Min-Max
Optimization) which recasts this combinatorial problem as a continuous minimax optimization problem and solves it by proximal gradient with a Max-Oracle.
M 3 O can also be applied to a more general scenario where we have missing entries in
M o and multiple groups of data with distinct unknown correspondence.
Experiments on simulated data, the MovieLens 100K dataset and Yale B database show that M 3 O achieves state-of-the-art performance over several baselines and can recover the ground-truth correspondence with high accuracy
I NTRODUCTION
In the era of big data, one usually needs to utilize data gathered from multiple disparate platforms when accomplishing a speciﬁc task.
However, the correspondence among the data samples from these different sources are often unknown due to either missing identity information or privacy reasons (Unnikrishnan et al., 2018; Gruteser et al., 2003; Das & Lee, 2018).
Examples include the multi-image matching problem studied in (Ji et al., 2014; Zeng et al., 2012;
Zhou et al., 2015), the record linkage problem (Chan & Loh, 2001) and the federated recommender system (Yang et al., 2020).
In the simplest scenario, we have two data matrices A
= [ a 1 ; :::; a n ] ⊤ , B =
[ b 1 ; :::; b n ] ⊤ with a i 2 R m A and b i 2 R m B , which are from two different platforms (data sources).
As discussed above, the correspondence ( a i ; b i ) may not be available, and thereby the goal is to recover the underlying correspondence between a 1 ; :::; a n and b ~ (cid:25) (1) ; :::; b ~ (cid:25) ( n ) , where ~ (cid:25) ( (cid:1) ) denotes an unknown permutation.
We can translate such problem described above as a matrix recovery problem, i.e., to recover the matrix M =
[ A; B ] from the permuted observation M o =
[ A; ~ PB ] , where ~ P 2 P n is an unknown permutation matrix and P n denotes the set of all n (cid:2) n permutation matrices.
We term this problem as M atrix R ecovery with U nknown C orrespondence ( MRUC ).
Inspired by the classical low-rank model for matrix recovery (Wright & Ma, 2021; Mazumder et al., 2010; Hastie et al., 2015), we especially focus on the scenario where the matrix M features a certain low-rank structure.
Such low-rank model has achieved great success in many applications like the recommender system (Schafer et al., 2007; Mazumder et al., 2010) and the image recovery and alignment problem (Zeng et al., 2012;
Zhou et al., 2015).
By denoting B o = ~ PB , we want to solve the following rank minimization problem for MRUC,
[Equation]
Practical applications.
It is known that the recommender system often suffers from data sparsity (Zhang et al., 2012) because users typically only provide ratings for very few items.
To enlarge the set of observable ratings for each user, we may harness extra data from multiple platforms (Netﬂix,
Amazon, Youtube, etc.).
One classical work on this problem is the multi-domain recommender system considered in (Zhang et al., 2012).
Unfortunately, their work neglects a crucial issue that data from these diverse platforms (or domains) are not always well aligned for two primary reasons.
The ﬁrst is that the same user may use different identities, or even leave nothing about their identities, on these platforms.
Another reason is that, those platforms are not allowed to share with each other the identity information about their users for preserving privacy.
Another application is the visual permutation learning problem (Santa Cruz et al., 2017), where one needs to recover the original image from the shufﬂed pixels.
Both of the two applications give rise to a challenging extension of the MRUC problem, where we not only need to recover multiple correspondence across different data sources, but also face the difﬁculty of dealing with the missing values in data matrix .
Relationship to the multivariate unlabeled sensing problem.
Problem (1) is closely related to the M ultivariate U nlabeled S ensing ( MUS ) problem, which has been studied in (Pananjady et al., 2017; Zhang et al., 2019a;b; Zhang & Li, 2020; Slawski et al., 2020b;a).
Speciﬁcally, the MUS is the multivariate linear regression problem with unknown correspondence, i.e., it solves
[Equation]
where W 2 R m 2 (cid:2) m 1 is the regression coefﬁcient matrix, Y 2 R n (cid:2) m 1 and X 2 R n (cid:2) m 2 denotes the output and the permuted input respectively, and ∥(cid:1)∥ F is the matrix Frobenius norm.
In fact, a concurrent work (Yao et al., 2021) studies the same rank minimization problem as (1), but their approach is to solve it using the algorithm developed for MUS problem.
Despite of the similarity to the MUS problem, we remark that MRUC problem has it own distinct features and, as shown in
Section 4, the algorithm for the MUS algorithm can not be directly and effectively applied, especially when there are multiple unknown correspondence and missing entries to be considered.
Related works.
To the best of our knowledge, the concurrent and independent (Yao et al., 2021) is the only work that also considers the MRUC problem.
Theoretically, (Yao et al., 2021) showed that there exists an non-empty open subset U (cid:18) R n (cid:2) ( m 1 + m 2 ) , such that 8 M 2 U , solving (1) is bound to recover the original correspondence.
However, such results only prove its existence for the subset
U and do not provide a concrete characterization.
Regarding the algorithm design, (Yao et al., 2021) follows the idea of (Slawski et al., 2020b;a) and treats problem (1) heuristically as a MUS problem.
However, there are two main drawbacks in their algorithm that largely limit its practical value.
First, in the dense permutation scenario, it ignores the interaction among the shufﬂed columns and hence can not utilize the prior knowledge on the unknown permutation to have an improved performance;
Second, their method can not deal with data with missing values.
Contributions of this work.
Our contributions in this work lie in both theoretical and practical aspects.
Theoretically, we are the ﬁrst to rigorously study how the rank of the data matrix is perturbed by the permutation, and show that problem (1) can be used to recover a generic low-rank random matrix almost surely.
Besides, we also propose a nuclear norm minimization problem as a surrogate for problem (1).
The most important theoretical result in this work is that we provide a non-asymptotic analysis to bound the error of the nuclear norm minimization problem under a mild assumption.
Practically, we propose an efﬁcient algorithm M 3 O that solves the nuclear norm minimization problem, which overcomes the aforementioned two shortcomings in (Yao et al., 2021).
Notably, M 3 O works very well even for an extremely difﬁcult task, where we need to recover multiple unknown correspondence from the data that are densely permuted and contain missing values.
We remark that this is so far a challenging problem unexplored in the existing literature.
Outline.
For conciseness, we will ﬁrst study the MRUC problem with single unknown correspondence, and then show that the theoretical results and the algorithm can be readily extended to the more complicated scenarios.
We start with building the theoretical results for (1) and its convex relaxation in Section 2.
Then, the algorithm is developed in Section 3.
The simulation results are presented in Section 4 and the conclusions are drawn in Section 5.
Notations.
Given two matrices X; Y 2 R n (cid:2) m , we denote ⟨ X; Y ⟩ =
∑ ni =1 ∑ mj =1 X ij Y ij as the matrix inner product.
We denote X ( i ) as the i th row of the matrix X and X ( i; j ) as the element at the i th row and the j th column.
We denote 1 m 2 R m and 1 n (cid:2) m 2 R n (cid:2) m as the all-one vector and matrix, respectively, and I n be the n (cid:2) n identity matrix.
For (cid:11) 2 R m , (cid:12) 2 R n , we deﬁne the operator (cid:8) as (cid:11) (cid:8) (cid:12) =
(cid:11) 1 ⊤ n + 1 m (cid:12)
⊤ 2 R m (cid:2) n .
We denote ∥ (cid:1) ∥ (cid:3) as the nuclear norm for matrices.
For vectors, we denote ∥ (cid:1) ∥ 0 , ∥ (cid:1) ∥ 1 as the zero norm and 1-norm respectively.
M ATRIX R ECOVERY VIA A L OW - RANK M ODEL
How is the matrix rank perturbed by the row permutation?
To answer this fundamental question, we ﬁrst introduce the cycle decomposition of a permutation.
Deﬁnition 1 (Cycle decomposition of a permutation (Dummit & Foote, 1991)) .
Let S be a ﬁnite set and (cid:25) ( (cid:1) ) be a permutation on S .
A cycle ( a 1 ; :::; a n ) is a permutation sending a j to a j +1 for 1 (cid:20) j (cid:20) n (cid:0) 1 and a n to a 1 .
Then a cycle decomposition of (cid:25) ( (cid:1) ) is an expression of (cid:25) ( (cid:1) ) as a union of several disjoint cycles 1 .
It can be veriﬁed that any permutation on a ﬁnite set has a unique cycle decomposition (Dummit &
Foote, 1991).
Therefore, we can deﬁne the cycle number of a permutation (cid:25) ( (cid:1) ) as the number of disjoint cycles with length greater than 1, which is denoted as C ( (cid:25) ) .
We also deﬁne the non-sparsity of a permutation as the Hamming distance between it and the original sequence, i.e., H ( (cid:25) ) =
∑ s 2 S
I [ (cid:25) ( s ) ̸ = s ] .
It is obvious
that H ( (cid:25) ) > C ( (cid:25) ) if (cid:25) is not an identity permutation.
As a simple example, we consider the permutation (cid:25) ( (cid:1) ) that maps the sequence (1,2,3,4,5,6) to (3,1,2,5,4,6).
Now the cycle decomposition for it is (cid:25) ( (cid:1) )
= (132)(45)(6) , and C ( (cid:25) )
= 2 , H ( (cid:25) )
= 5 .
In all the following theoretical results, we denote the original matrix as M =
[ A; B ] 2 R n (cid:2) m with
A 2 R n (cid:2)
m A , B 2 R n (cid:2) m B , and rank( M ) = r , rank( A ) =
r A , rank( B ) =
r B .
We denote the corresponding permutation as (cid:25) P ( (cid:1) ) for any permutation matrix P 2 P n .
The following proposition says that the perturbation effect of a permutation (cid:25) on the rank of M becomes stronger, if (cid:25) permutes more rows and contains less cycles.
Proposition 1.
8 P 2 P n , we have
[Equation]
We have similar result for the case with multiple permutations, which is summarized in Corollaryin Appendix A.1.
It turns out that, without any assumption on M , (3) is the tightest upper bound for the rank of a perturbed matrix.
Notably, the following proposition says that the upper bound in (3) is attained with probability 1 for a generic low-rank random matrix.
Deﬁnition 2.
A probability distribution on R is called a proper distribution if its density function p ( (cid:1) ) is absolutely continuous with respect the Lebesgue measure on R .
Proposition 2.
If the original matrix M is a random matrix with M = RE where R 2 R n (cid:2) r and
E 2 R r (cid:2)
m are two random matrices whose entries are i.i.d and follow a proper distribution on R , and r (cid:20)
min f √ n 2 ; m A ; m B g , then 8 P 2 P n , the equality
[Equation]
holds with probability 1.
Convex relaxation for the rank function.
Despite the previous theoretical justiﬁcation for problem (1), it is non-convex and non-smooth.
Another crucial issue is that we often have a noisy observation matrix and it is well known that the rank function is extremely sensitive to the additive noise.
In this paper, we assume that the observation matrix is corrupted by i.i.d Gaussian additive noise, i.e., M o =
[ A o ; B o ] =
[ A; ~ PB ] + W; where W ( i; j ) (cid:24) N (0 ; (cid:27) 2 ) ; where (cid:27) 2 reﬂects the strength of the noise.
We ﬁrst denote the singular values of a matrix X 2 R n (cid:2) m as (cid:27) 1 X ; :::; (cid:27) kX where k = min
f n; m g .
Since rank ( X ) =
∥
[ (cid:27) 1 X ; :::; (cid:27) kX ] ∥ 0 , from Proposition 2 we can view the perturbation effect of a permutation to a low-rank matrix as breaking the sparsity of its singular values.
This view leads naturally to the well-known 1-norm minimization problem which has been proven robust to additive noise and can yield a sparse solution (Wright & Ma, 2021), i.e.,
[Equation]
Since for an arbitrary matrix, the 1-norm of its singular values is equivalent to its nuclear norm, we refer problem (5) as the nuclear norm minimization problem.
Theoretical justiﬁcation for the nuclear norm.
Nuclear norm has a long history used as a convex surrogate for the rank, and it has been theoretically justiﬁed for applications like low-rank matrix completion (Candès & Tao, 2010; Wright & Ma, 2021).
It is also important to see whether the nuclear norm is still a good surrogate for the rank minimization problem (1).
In this work, we establish a sufﬁcient condition on A and B under which problem (5) is provably justiﬁed for correspondence recovery.
We denote A =
∑ r A i =1 (cid:27) iA u iA v
i ⊤ A ; B =
∑ r B i =1 (cid:27) iB u iB v i ⊤ B as the singular values decomposition of A and B , where the (cid:27) iA and (cid:27) iB are the non-zero singular values.
Firstly, from the deﬁnition of nuclear norm, it can be simply veriﬁed for any P 2 P n that
[Equation]
) where we denote N = max f∥ A ∥ (cid:3) ; ∥ B ∥ (cid:3) g and Z = min f∥ A ∥ (cid:3) ; ∥ B ∥ (cid:3) g .
The inequality (6) indicates that A and B should have comparable magnitude, i.e., ∥ A ∥ (cid:3) (cid:25)
∥
B ∥ (cid:3) , otherwise the inﬂuence of the permutation will be less signiﬁcant.
Therefore, we are interested in the scenario where the singular values of A and B are comparable, which is described as the following Assumption 1.
Assumption 1.
There exists a constant ϵ 1 (cid:21)
such that
[Equation]
where we denote that (cid:27) iA =
if i > r A , and (cid:27) iB =
if i > r B .
Similar to the matrix rank, we also need a proper low-rank assumption on the matrix M for the nuclear norm.
In this work, we particularly study the scenario that the left singular vectors of A and
B are similar, which we formally describe as Assumption 2.
We refer Assumption 2 as a proper low-rank assumption, because it indicates that the column space of M can be approximated by the column space of one of its submatrices.
Assumption 2.
There exists a constant ϵ 2 (cid:21)
such that
[Equation]
where we denote T =
min f r A ; r B g .
Furthermore, we also need that all the column singular vectors u 1 A ; :::; u TA ; u 1 B ; :::; u TB are variant under any P 2 P n with P ̸
= I n : we deﬁne a vector u 2 R n to be variant under a P 2 P n if
Pu ̸ = u .
One simple and weak condition for a vector u to satisfy such property is that u dose not contains duplicated elements, which leads to the following Assumption 3.
Assumption 3.
There exists a constant ϵ 3 (cid:21)
such that
[Equation]
where U = f u 1 A ; :::; u TA ; u 1 B ; :::; u TB g .
In summary, the assumptions mentioned above feature a typical low-rank structure in M , and implies that the nuclear norm of M is sensitive to permutation.
With the three assumptions, we have the following important theorem, which provides high probability bound for the approximation error of (5).
We denote the solution to (5) as P (cid:3) , and let (cid:25) (cid:3) and ~ (cid:25) be the corresponding permutation to the permutation matrices P (cid:3)⊤ and ~ P , respectively.
We deﬁne the difference between the two permutations (cid:25) (cid:3) and ~ (cid:25) as the Hamming distance
[Equation]
Theorem 1.
Under Assumptions 1, 2 and 3, if additionally ϵ 1 (cid:20) M 4 r , ϵ 2 (cid:20) min f 1 2 p 2 T ; p 2 M 2 N g , and
[Equation]
M 16 L 2 , then the following bound for the Hamming distance
[Equation]
holds with probability at least 1 (cid:0) 2 exp f(cid:0) D 8 L(cid:27) g , where L = max f n; m g , D = ∥
A ∥ (cid:3) + ∥ B ∥ (cid:3) .
The proof to all the aforementioned theoretical results are provided in Appendix A.1.
Remark 1.
From Theorem 1 we can see that when ϵ 3 > 0 , and ϵ 1 ! 0 , ϵ 2 ! 0 , (cid:27) !
0 , the error d H ( (cid:25) (cid:3) ; ~ (cid:25) ) will converge to zero with probability 1.
Furthermore, we can also discover that the correspondence can be difﬁcult to recover when:
(cid:15)
The rank of original matrix M is high, which can be seen from (10).
(cid:15)
The magnitude of A and B w.r.t rank or nuclear norm are not comparable, which can be seen from (6) and (7).
(cid:15)
The strength of noise is high, which can be seen from (10) and the probability in Theorem 1.
Notably, the numerical experiments in Section 4.1 corroborate our claims as well.
Remark 2.
Additionally, from the proof of Theorem 1 we ﬁnd that the fundamental reason for the success of (5) is that if M satisﬁes the previous assumptions, we have
[Equation]
In many applications, we can only observe part of the full data.
Therefore, it is also worthwhile to investigate whether (11) still holds when we can only access a small subset of the entries in M o .
Notably, Figure 1 gives the positive answer and shows that the relationship (11) is gracefully degraded when the percentage of observable entries is decreasing.
This phenomenon is remarkable since it indicates the original correspondence can be recovered from only part of the full data.
The matrices used to generate Figure 1 are the same as those in
Section 4.1, and the nuclear norm is computed approximately by ﬁrst ﬁlling the missing entries using Soft-Impute algorithm (Mazumder et al., 2010).
A LGORITHM
In this section, we consider the scenario with missing values, i.e., our observed data is P Ω ( M o ) =
P Ω ([ A o ; B o ]) , where P Ω is an operator that selects entries that are in the set of observable indices Ω .
In this scenario, problem (5) can not be directly used since the evaluation of the nuclear norm and optimization of the permutation are coupled together.
Inspired by the matrix completion method (Hastie et al., 2015; Mazumder et al., 2010), we propose to solve an alternative form of (5) as follows,
[Equation] [Equation]
where (cid:21) > 0 is the penalty coefﬁcient.
We denote that c M =
[ c M A ; c M B ] and c M A ; c M B are the two submatrices with the same dimension as A o and B o respectively.
We can write (12) equivalently as
[Equation]
[Equation]
where C ( c M B ) 2 R n (cid:2) n is the pairing cost matrix with
[Equation]
Baseline algorithm.
A conventional strategy to handle an optimization problem like (13) is the alternating minimization or the block coordinate descent
algorithm
(Abid et al., 2017).
Speciﬁcally, it executes the following two updates iteratively until it converges.
[Equation]
The ﬁrst update step (14) is a convex optimization problem and can be solved by the proximal gradient
algorithm (Mazumder et al., 2010).
The second update step (15) is actually a discrete optimal transport problem which can be solved by the classical Hungarian algorithm with time complexity O ( n 3 ) (Jonker & Volgenant, 1986).
However, as we will see in the Section 4, this algorithm performs poorly, and it is likely to fall into an undesirable local solution quickly in practice.
Speciﬁcally, the main reason is that the solution of (15) is often not unique and a small change in c
M B would lead to large change of b P .
To address this issue, we propose a novel and efﬁcient algorithm M 3 O algorithm based on the entropic optimal transport (Peyré et al., 2019) and min-max optimization (Jin et al., 2020).
Smoothing the permutation with entropy regularization.
For any a 2 R n ; b 2 R m , we deﬁne (cid:5)
[Equation]
which is also known as the Birkhoff polytope.
The famous Birkhoff-von Neumann theorem (Birkhoff, 1946) states that the set of extremal points of (cid:5)( 1 n ; 1 n ) is equal to P n .
Inspired by (Xie et al., 2021) and the interior point method for linear programming (Bertsekas, 1997), in order to smooth the optimization process of the baseline algorithm, we relax P from being an exact permutation matrix, i.e., to keep P staying inside the Birkhoff polytope (cid:5)( 1 n ; 1 n ) .
That is, we propose to replace the combinatorial problem (15) with the following continuous optimization problem
[Equation]
where H ( P ) def.
=
∑ i;j P ( i; j )(log( P ( i; j )) (cid:0) 1) is the matrix negative entropy and ϵ > 0 is the regularization coefﬁcient.
Notably, (16) is also known as the Entropic Optimal Transport (EOT) problem. According to (Peyré et al., 2019), (16) is a strongly convex optimization problem with respect to 1-norm and can be solved roughly in the O ( n 2 ) complexity per iteration by the Sinkhorn algorithm.
Speciﬁcally, the Sinkhorn algorithm solves the dual problem of (16),
[Equation]
which reduces the variables dimension from n 2 to 2 n and is thus greatly favorable in the high dimension scenario.
By substituting the inner minimization problem of (13) with (16), we end up with solving the following unconstrained min-max optimization problem
[Equation]
Follows the idea of (Jin et al., 2020), we consider to adopt a proximal gradient algorithm with a
Max-Oracle for (18).
Speciﬁcally, we employ the Skinhorn algorithm (Peyré et al., 2019) as the MaxOracle to retrieve an " -good solution of the inner max problem (17).
We summarize our proposed algorithm M 3 O ( M atrix recovery via M in- M ax O ptimization) in Algorithm 1, where prox (cid:21) ∥(cid:1)∥ (cid:3) ( (cid:1) ) is the proximal operator of nuclear norm and (cid:26) k is the gradient stepsize.
The convergence property of M 3 O can be obtained by following (Jin et al., 2020), which shows that, with a decaying stepsize,
M 3 O is bound to converge to an " -good Nash equilibrium within O ( " (cid:0) 2 ) iterations.
Algorithm 1 while not converged do
For the tolerance " , run the Sinkhorn algorithm to ﬁnd (cid:11) (cid:3) , (cid:12) (cid:3) such that
[Equation]
)) ; where
[Equation]
4 end
Remark 3.
A recent work (Xie et al., 2020) proposes a decaying strategy for the entropy regularization coefﬁcient ϵ in (16) so that the optimal solutions of (15) and (16) do not deviate too much.
Inspired by it, in our practice, we take large ϵ in the beginning and gradually shrink it by half until the objective function stops improving for K steps.
Remark 4.
A useful trick is that we should not take large stepsize (cid:26) k in the early iterations because the permutation matrix could still be far away from the optimal one.
However, a small stepsize would lead to slow convergence.
Heuristically, we propose an adaptive stepsize strategy that performs well in practice.
For the solution of (16) b P k at the k th iteration, we compute the two statistics
[Equation]
Here (cid:14) k represents how fast the permutation matrix b P k changes over the iterations, while c k measures how far the current b P k is close to an exact permutation matrix.
Both (cid:14) k and c k reﬂect the conﬁdence on the current found correspondence.
Based on them, we set the stepsize as (cid:26) k +
=
(1 (cid:0) (cid:14) k )(1 (cid:0) c k ) !
; where ! > 0 is a tunable parameter which is often set to a value between 0.5 to 3. !
actually trades off the convergence speed and ﬁnal performance.
The smaller the ! , the faster the convergence.
Therefore, a practical way is to start with a small ! , and gradually increase it until the ﬁnal performance stops improving.
Remark 5.
As discussed in Section 1, in many cases we have to deal with the problem that involves multiple correspondence, i.e., we need to recover the matrix M
=
[ A; B 1 ; :::; B d ] from the observation data P Ω ( M o ) , where
[Equation]
where ~ P l 2 P n and W is a noise matrix.
We refer such problem as the d -correspondence problem.
An important observation is that, although the number of possible correspondence increase exponentially as d grows, the complexity of M 3 O per iteration only linearly increases with d and can be implemented in a fully parallel fashion.
Speciﬁcally, in this scenario, we solve the problem
[Equation]
d
[Equation]
where we denote c M =
[ c M A ; c M B 1 ; :::; c M B d ] .
Here c M A and c M B l have the same dimension with A o and B lo , respectively.
One can ﬁnd that the inner problems for solving P l are actually decoupled for each l , which guarantees an efﬁcient parallel implementation.
Remark 6.
Since x problem (12) has a similar form to that considered in (Mazumder et al., 2010).
We adopt the same tuning strategy of (cid:21) as in (Mazumder et al., 2010), which suggests that we should start with large (cid:21) and gradually decrease it.
We relegate more details about M 3 O to Appendix A.6.
E XPERIMENTS
[Figure]
In this section, we evaluate our proposed
M 3 O on both synthetic and real-world datasets, including the MovieLens 100K and the Extended Yale B dataset.
We also provide an ablation study for the decaying entropy regularization strategy and the adaptive stepsize strategy proposed in Remarks 3 and 4.
In all the experiments, we employ the Soft-Impute algorithm (Mazumder et al., 2010) as a standard algorithm for matrix completion.
Extra experiment details and auxiliary results can be found in Appendix A.9.
Algorithms.
We denote the following algorithms for comparison in all the experiments:
Oracle : Running the Soft-Impute algorithm with ground-truth correspondence.
Baseline : The Baseline algorithm in (14) and (15).
MUS : Since there is currently no existing algorithm directly applicable to the scenario considered by (19), inspired by (Yao et al., 2021), we modify and extend the algorithm in (Zhang & Li, 2020), which is originally proposed for the MUS problem, to deal with the MRUC problem.
The details of the adapted algorithm are provided in Appendix A.
4. S YNTHETIC DATA
We ﬁrst investigate the property of our proposed M 3 O algorithm on the synthetic data.
Data generation.
We generate the original data matrix in this form M = RE + (cid:17)W; where
R 2 R n (cid:2) r , E 2 R r (cid:2) m , W 2 R n (cid:2) m and (cid:17) > 0 indicates the strength of the additive noise.
The entries of R , E , W are all i.i.d sampled from the N (0 ; 1) .
Then we split the data matrix
M by M =
[ A; B 1 ; :::; B d ] where we denote A 2 R n (cid:2) m A , B 1 2 R n (cid:2) m 1 , ..., B d 2 R n (cid:2) m d to represent data from d + 1 data sources.
The permuted observation matrix M o is obtained by ﬁrst generating d permutation matrices P 1 ; :::; P d randomly and independently, and then computing
M o =
[ A; P 1 B 1 ; :::; P d B d ] .
Finally, we remove (1 (cid:0) j Ω j (cid:1) 100% =
( n (cid:1) m )) percent of the entries of
M o randomly and uniformly, where j Ω j indicating the number of observable entries.
Ablation study.
We denote the following variants of M 3 O for the ablation study.
M 3 O-AS-DE : M 3 O with both Adpative Stepsize and Decaying Entropy regularization.
M 3 O-DE : M 3 O with Decaying Entropy regularization only.
M 3 O-DE-1 and M 3 O-DE-2 adopt constant stepsize (cid:26) k = 0 : 5 and (cid:26) k = 0 : 01 , respectively.
M 3 O-AS : M 3 O with Adpative Stepsize only.
The entropy coefﬁcient ϵ is ﬁxed to 0.0005.
In the following results, we denote (cid:25) l as the corresponding permutation to P l .
We initialize c M from
Gaussian distribution for the M 3 O algorithm and its variants.
We choose initial ϵ as 0.1 and K =as the default for the decaying entropy regularization, and set !
=
3 as the default for the adaptive stepsize.
We also report the achieved objective values of (19) for the tested algorithms, except for the MUS algorithm since it has a different objective.
We denote ^ (cid:25) as the recovered permutation.
Results.
Figure 2 displays the result under the setting (cid:17) = 0 : 1 , j Ω j (cid:1) 100% =
( n (cid:1) m )
= 80% , n = m = 100 , r = 5 , d = 1 , m A = 60 and m 1 = 40 .
The algorithm M 3 O-AS-DE achieves the best result, and can recover the ground-truth correspondence.
M 3 O-AS behaves similarly to Baseline and MUS.
They all converge to a poor local solution quickly.
M 3 O-DE-1 converges quickly and also falls into a poor local solution due to large stepsize, while M 3 O-DE-2 adopts a small stepsize and hence suffers from slow convergence.
Due to the superiority of M 3 O-AS-DE over the other variants, in the following results, we refer M 3 O as M 3 O-AS-DE for short.
Figure 3 examine M 3 O on a 1-correspondence problem under different regimes w.r.t j Ω j , (cid:17) , r and m A =n .
Here we use m A =n to control the difference of the magnitude of the submatrices.
As we can see, the results are well aligned with our prediction in Remarks 1 and 2.
Finally, we examine M 3 O on a few d-correspondence problems.
See Table 1 for various results, where we set r = 5 and " = 0 : 1 .
Notice that for the 4-correspondence problem in the table, there are (100!)
4 possible correspondence.
Even for such a difﬁcult problem, M 3 O is able to recover 61.5% of the ground-truth correspondence with a good initialization.
[Figure]
The default setting is j Ω j (cid:1) 100% =
( n (cid:1) m )
= 80% , (cid:17)
= 0 : 1 , n = m = 100 , r = 5 , m A = 60 , and m 1 = 40 .
The mean with minimum and maximum are calculated from 10 different random initializations.
[Table]
4. M ULTI - DOMAIN RECOMMENDER SYSTEM WITHOUT CORRESPONDENCE
In this section
[Table]
we study the performance of M 3 O on a real world dataset MovieLens 100K 2 , which is a widely used movie recommendation dataset (Harper & Konstan, 2015).
In this application, we mainly focus on the metric Root Mean Squared Error (RMSE),
i
[Table]
e.,
[Equation]
Data.
MovieLens 100K contains 100,000 ratings within the scale 1-5.
The ratings are given by 943 users on 1,682 movies.
Genre information about movies is also provided.
We adopt a similar setting with (Zhang et al., 2012).
We extract ﬁve most popular genres, which are Comedy, Romance,
Drama, Action, Thriller respectively, to deﬁne the data from 5 different domains (or platforms).
In addition to (Zhang et al., 2012), we randomly permute the indexes of the users from these ﬁve domains respectively, so that the correspondence among these data become unknown.
In this way, the problem belongs to the 4-correspondence problem as discussed before.
The ratings are split randomly, with 80% of them as the training data and the other 20% of them as the test data.
Algorithms.
We consider the following additional algorithms for comparison.
SIC :
Running the Soft-Impute algorithm independently for the 5 different platforms.
SIR : Running the Soft-Impute algorithm with Randomly generated correspondence.
Results.
As discussed in experiments on the simulated data, the exact recovery of correspondence becomes impossible due to the small amount of observable entries.
Therefore, in the following experiment, since exact correspondence is not needed, we ﬁx ϵ = 0 : 05 for M 3 O. Table 2 shows the results by averaging the RMSE on the test data over 10 different random seeds.
We can ﬁrst see that the matrix completion with a wrong correspondence, i.e., SIR, can be harmful to the overall performance since it is even worse than the results of SIC.
Notably, although the ground-truth correspondence can not be recovered, each platform can still beneﬁt from M 3 O since it improves the performance over SIC.
This is mainly because M 3 O is still able to correspond similar users for inferring missing ratings.
On the contrary, since both Baseline and MUS can only establish an exact one-to-one correspondence for each user, they fail to improve SIC signiﬁcantly.
Remarkably, M 3 O is only inferior to the Oracle method a little, and even achieves lower test RMSE than the
Oracle method on the Comedy genre.
[Table]
[Figure]
We show that M 3 O is ﬂexible and can also be used to recover matrix that is not in the form [ A; PB ] .
We can see this from the problem formulation in (13), where the cost matrix C ( (cid:1) ) can be constructed in other ways as long as it is a function of a permutation.
Typically, M 3 O can be used to solve a challenging face image recovery problem.
The original face image with size 180 (cid:2) 180 in Figure 4(a) comes from the Extend Yale
B database (Georghiades et al., 2001).
The corrupted image is visualized in Figure 4(b), where the pixel blocks with size 30 (cid:2) 30 in the upper left are shufﬂed randomly, and 30% of the total pixels are removed.
This kind of problem is recently considered in (Santa Cruz et al., 2017), which proposes to recover the corrupted image in a data-driven way using convolutional neural networks.
However, we show that it is possible to recover the image without additional data by merely exploiting the underlying low-rank structure of the image itself.
This experiment setting is similar to that in (Yao et al., 2021) but the algorithm in (Yao et al., 2021) can not be applied since it can not work with the missing values.
The MUS algorithm is also not applicable since this problem can not be written in the form of linear regression problem.
From
Figure 4(c) and 4(d)
we can ﬁnd that M
3 O performs better than the Baseline, and can even recover the original orders of pixel blocks.
More results similar to the Figure 4 and experiment details are provided in Appendix A.9.
C ONCLUSION
In this paper, we have studied the important MRUC problem where part of the observed submatrix is shufﬂed.
This problem has not been well explored in the existing literature.
Theoretically, we are the ﬁrst to rigorously analyze the role of low-rank model in the MRUC problem, and is also the ﬁrst to show that minimizing nuclear norm is provably efﬁcient for recovering a typical low-rank matrix.
For practical implementations, we propose a highly efﬁcient algorithm, the M 3 O algorithm, which is shown to consistently achieve the best performance over several baselines in all the tested scenarios.
It is worthwhile to point out that apart from the two applications we have studied in this paper, this problem could arise in more scenarios like the gnome assembly problem (Huang & Madan, 1999), the video pose tracking problem (Ganapathi et al., 2012) and the privacy-aware sensor networks (Gruteser et al., 2003), etc.
We believes that our work provides a general framework to deal with unknown correspondence issue in these scenarios.
As we have shown in Figure 3, one major limit of our algorithm is the sensitivity to the initialization.
The phenomenon is exacerbated when the additive noise is high or the numbers of observable entries are small.
We suggest to try with a few different initialization strategy when applying M 3 O to a speciﬁc task.
Finding stable initialization strategy is also an important task for our future works.