A H YPERGRADIENT A PPROACH TO R OBUST R EGRESSION WITHOUT C ORRESPONDENCE
A BSTRACT
We consider a regression problem, where the correspondence between the input and output data is not available.
Such shufﬂed data are commonly observed in many real world problems.
Take ﬂow cytometry as an example: the measuring instruments are unable to preserve the correspondence between the samples and the measurements.
Due to the combinatorial nature of the problem, most of the existing methods are only applicable when the sample size is small, and are limited to linear regression models.
To overcome such bottlenecks, we propose a new computational framework — ROBOT — for the shufﬂed regression problem, which is applicable to large data and complex models.
Speciﬁcally, we propose to formulate regression without correspondence as a continuous optimization problem.
Then by exploiting the interaction between the regression model and the data correspondence, we propose to develop a hypergradient approach based on differentiable programming techniques.
Such a hypergradient approach essentially views the data correspondence as an operator of the regression model, and therefore it allows us to ﬁnd a better descent direction for the model parameters by differentiating through the data correspondence.
ROBOT is quite general, and can be further extended to an inexact correspondence setting, where the input and output data are not necessarily exactly aligned.
Thorough numerical experiments show that ROBOT achieves better performance than existing methods in both linear and nonlinear regression tasks, including real-world applications such as ﬂow cytometry and multi-object tracking.
I NTRODUCTION
Regression analysis has been widely used in various machine learning applications to infer the the relationship between an explanatory random variable (i.e., the input) X ∈ R d and a response random variable (i.e., the output) Y ∈ R o (Stanton, 2001).
In the classical setting, regression is used on labeled datasets that contain paired samples { x i , y i } ni =1 , where x i , y i are realizations of X , Y , respectively.
Unfortunately, such an input-output correspondence is not always available in some applications.
One example is ﬂow cytometry, which is a physical experiment for measuring properties of cells, e.g., afﬁnity to a particular target (Abid & Zou, 2018).
Through this process, cells are suspended in a ﬂuid and injected into the ﬂow cytometer, where measurements are taken using the scattering of a laser.
However, the instruments are unable to differentiate the cells passing through the laser, such that the correspondence between the cell proprieties (i.e., the measurements) and the cells is unknown.
This prevents us from analyzing the relationship between the instruments and the measurements using classical regression analysis, due to the missing correspondence.
Another example is multi-object tracking, where we need to infer the motion of objects given consecutive frames in a video.
This requires us to ﬁnd the correspondence between objects in the current frame and those in the next frame.
The two examples above can be formulated as a shufﬂed regression problem.
Speciﬁcally, we consider a multivariate regression model
[Equation]
where X ∈ R d , Z ∈ R e are two input vectors,
Y ∈ R o is an output vector, f : R d + e → R o is an unknown regression model with parameters w , and ε is a random noise independent on X and Z .
When we sample realizations from such a regression model, the correspondence between ( X, Y ) and
Z is not available.
Accordingly, we collect two datasets D 1 = { x i , y i } ni =1 and D 2 = { z j } nj =1 , and there exists a permutation π ∗ such that ( x i , z π ( i ) ) corresponds to y i in the regression model.
Our goal is to recover the unknown model parameter w .
Existing literature also refer to the shufﬂed regression problem as unlabeled sensing , homomorphic sensing , and regression with an unknown permutation (Unnikrishnan et al., 2018).
Throughout the rest of the paper, we refer to it as Regression WithOut
Correspondence (RWOC).
A natural choice of the objective for RWOC is to minimize the sum of squared residuals with respect to the regression model parameter w up to the permutation π ( · ) over the training data, i.e.,
[Equation]
Existing works on RWOC mostly focus on theoretical properties of the global optima of equation 1 in estimating w and π (Pananjady et al., 2016; 2017b; Abid et al., 2017;
Elhami et al., 2017; Hsu et al., 2017; Unnikrishnan et al., 2018; Tsakiris & Peng, 2019).
The development of practical algorithms, however, falls far behind from the following three aspects:
• Most of the works are only applicable to linear regression models.
• Some of the existing algorithms are of very high computational complexity, and can only handle small number of data points in low dimensions (Elhami et al., 2017; Pananjady et al., 2017a; Tsakiris et al., 2018; Peng & Tsakiris, 2020).
For example, Abid & Zou (2018) adopt an Expectation Maximization (EM) method where Metropolis-Hastings sampling is needed, which is not scalable.
Other algorithms choose to optimize with respect to w and π in an alternating manner, e.g., alternating minimization in Abid et al.
However, as there exists a strong interaction between w and π , the optimization landscape of equation 1 is ill-conditioned.
Therefore, these algorithms are not effective and often get stuck in local optima.
• Most of the works only consider the case where there exists an exact one-to-one correspondence between D 1 and D 2 .
For many more scenarios, however, these two datasets are not necessarily well aligned.
For example, consider D 1 and D 2 collected from two separate databases, where the users overlap, but are not identical.
As a result, there exists only partial one-to-one correspondence.
There is a similar situation in multiple-object tracking: Some objects may leave the scene in one frame, and new objects may enter the scene in subsequent frames.
Therefore, not all objects in different frames can be perfectly matched.
The RWOC problem with partial correspondence is known as robustRWOC, or rRWOC (Varol & Nejatbakhsh, 2019), and is much less studied in existing literature.
To address these concerns we propose a new computational framework — ROBOT (Regression withOut correspondence using Bilevel OptimizaTion).
Speciﬁcally, we propose to formulate regression without correspondence as a continuous optimization problem.
Then by exploiting the interaction between the regression model and the data correspondence, we propose to develop a hypergradient approach based on differentiable programming techniques (Duchi et al., 2008; Luise et al., 2018).
Our hypergradient approach treats the data correspondence as an operator of the regression, i.e., for a given w , the optimal correspondence is
[Equation]
Accordingly, when applying gradient descent to (1), we need to ﬁnd the gradient with respect to w by differentiating through both the objective function L and the data correspondence (cid:98) π ( w ) .
For simplicity, we refer to such a gradient as “hypergradient”.
Note that due to its discrete nature, (cid:98) π ( w ) is actually not continuous in w .
Therefore, the above-mentioned hypergradient does not exist.
To address this issue, we further propose to construct a smooth approximation of (cid:98) π ( w ) by adding an additional regularizer to equation 2, and then we replace (cid:98) π ( w ) with our proposed smooth replacement when computing the hypergradient of w .
Moreover, we also propose an efﬁcient and scalable implementation of hypergradient computation based on simple ﬁrst order algorithms and implicit differentiation, which outperforms conventional automatic differentiation in terms of time and memory cost.
ROBOT can also be extended to the robust RWOC problem, where D 1 and D 2 are not necessarily exactly aligned, i.e., some data points in D 1 may not correspond to any data point in D 2 .
To tackle such a problem, we relax the constraints on the permutation π ( · ) (Liero et al., 2018) to automatically match related data points and ignore the unrelated ones.
We conduct thorough numerical experiments to demonstrate the effectiveness of ROBOT.
For
RWOC (i.e., exact correspondence), we use several synthetic regression datasets and a real-world gated ﬂow cytometry dataset, and we show that ROBOT outperforms baseline methods by signiﬁcant margins.
For robust RWOC (i.e., inexact correspondence), in addition to synthetic datasets, we consider a vision-based multiple-object tracking task, and we show that ROBOT achieves signiﬁcant improvement over baseline methods.
Notations .
Let (cid:107)·(cid:107) 2 denote the (cid:96) 2 norm of vectors, (cid:104)· , ·(cid:105) the inner product of matrices, i.e., (cid:104) A, B (cid:105) =
(cid:80) i,j A ij B ij for matrices A and B .
a i : j are the entries from index i to index j of vector a .
Let 1 n denote an n -dimensional vector of all ones.
Denote d ( · ) d ( · ) the gradient of scalars, and ∇ ( · ) ( · ) the
Jacobian of tensors.
We denote [ v 1 , v 2 ] the concatenation of two vectors v 1 and v 2 .
N ( µ, σ 2 ) is the
Gaussian distribution with mean µ and variance σ 2 .
ROBOT: A H YPERGRADIENT A PPROACH FOR RWOC
In this section, we develop our hypergradient approach for RWOC.
We ﬁrst introduce a continuous formulation equivalent to (1), and then we propose a smooth bi-level relaxation with an efﬁcient hypergradient descent algorithm.
2. E QUIVALENT C ONTINUOUS F ORMULATION
We propose a continuous optimization problem equivalent to (1).
Speciﬁcally, we rewrite an equivalent form of (1) as follows,
[Equation]
where P denotes the set of all n × n permutation matrices, C ( w )
∈ R n × n is the loss matrix with
[Equation]
Note that P is the discrete feasible set of the inner minimization problem of (3). Without affecting the optimality, we can relax S ∈ P to a convex set, as suggested by the next theorem.
Proposition 1.
Given any a ∈ R n and b ∈ R m , we deﬁne
[Equation]
The optimal solution to the inner discrete minimization problem of (3) is also the optimal solution to the following continuous optimization problem,
[Equation]
This is a direct corollary of the Birkhoff-von Neumann theorem (Birkhoff, 1946; Von Neumann, 1953), and we refer to Appendix A for more details.
Theorem 1 allows us to replace P in (3) with Π( 1 n , 1 n ) , which is also known as the Birkhoff polytope 1 (Ziegler, 2012).
Accordingly, we obtain the following continuous formulation,
[Equation]
Remark 1.
In general, (3) can be solved by linear programming algorithms (Dantzig, 1998).
2. C ONVENTIONAL W ISDOM : A LTERNATING M INIMIZATION
Conventional wisdom for solving (5) suggests to use alternating minimization (AM, Abid et al.
Speciﬁcally, at the k -th iteration, we ﬁrst update S by solving
[Equation]
and then given S ( k ) , we update w using gradient descent or exact minimization, i.e.,
[Equation]
However, AM works poorly for solving (5) in practice.
This is because w and S have a strong interaction throughout the iterations:
A slight change to w may lead to signiﬁcant change to S .
Therefore, the optimization landscape is ill-conditioned, and AM can easily get stuck at local optima.
2. S MOOTH B I - LEVEL R ELAXATION
To tackle the aforementioned computational challenge, we propose a hypergradient approach, which can better handle the interaction between w and S .
Speciﬁcally, we ﬁrst relax (5) to a smooth bilevel optimization problem, and then we solve the relaxed bi-level optimization problem using the hypergradient descent algorithm.
We rewrite (5) as a smoothed bi-level optimization problem,
[Equation]
where H ( S ) = (cid:104) log S, S (cid:105) is the entropy of S .
The regularizer H ( S ) in (6) alleviates the sensitivity of S ∗ ( w ) to w .
Note that without such a regularizer, we solve
[Equation]
and the resulting S ∗ ( w ) can be discontinuous in w .
This is because S ∗ ( w ) is the optimal solution of a linear optimization problem, and usually lies on a vertex of Π( 1 n , 1 n ) .
This means that if we change w , S ∗ ( w ) either stays the same or jumps to another vertex of Π( 1 n , 1 n ) .
The jump makes
S ∗ ( w ) highly sensitive to w .
To alleviate this issue, we propose to smooth S ∗ ( w ) by adding an entropy regularizer to the lower level problem.
The entropy regularizer enforces S ∗ (cid:15) ( w ) to stay in the interior of Π( 1 n , 1 n ) , and S ∗ (cid:15) ( w ) changes smoothly with respect to w , as suggested by the following theorem.
Theorem 1.
For any (cid:15) > 0 , S ∗ (cid:15) ( w ) is differentiable, if the cost C ( w ) is differentiable with respect to w .
Consequently, the objective F (cid:15) ( w ) =
(cid:104) C ( w ) , S ∗ (cid:15) ( w ) (cid:105) is also differentiable.
The proof is deferred to Appendix C. Note that (6) provides us a new perspective to interpret the relationship between w and S .
As can be seen from (6), w and S have different priorities: w is the parameter of a leader problem, which is of a higher priority; S is the parameter of a follower problem, which is of a lower priority, and can also be viewed as an operator of w — denoted by
S ∗ (cid:15) ( w ) .
Accordingly, when we minimize (6) with respect to w using gradient descent, we should also differentiate through S ∗ (cid:15) .
We refer to such a gradient as “hypergradient”, deﬁned as follows,
[Equation]
We further examine the alternating minimization algorithm from the bi-level optimization perspective: Since ∇ w L ( w ( k − 1) , S ( k ) ) is not differentiable through S ( k ) , AM is essentially using an inexact gradient.
From a game-theoretic perspective 2 , (6) deﬁnes a competition between the leader w and the follower S .
When using AM, S only reacts to what w has responded.
In contrast, when using the hypergradient approach, the leader essentially recognizes the follower’s strategy and reacts to what the follower is anticipated to response through ∂ F
[Equation]
∗ (cid:15) ( w ) ∂w .
In this way, we can ﬁnd a better descent direction for w .
Remark 2.
We use a simple example of quadratic minimization to illustrative why we expect the bilevel optimization formulation in (6) to enjoy a benign optimization landscape.
We consider a quadratic function
[Equation]
) where a 1 ∈ R d 1 , a 2 ∈ R d 2 , a = [ a 1 , a 2 ] ,
P ∈ R ( d 1 + d 2 ) ×
( d 1 + d 2 ) , b ∈ R d
1 + d 2 .
Let
P = ρ 1 d 1 + d 2 1 (cid:62) d 1 + d 2 + (1 − ρ ) I
d 1 + d 2 , where I d 1 + d 2 is the identity matrix, and ρ is a constant.
We solve the following bilevel optimization problem,
[Equation]
where λ is a regularization coefﬁcient.
The next proposition shows that ∇ 2 F ( a 1 ) enjoys a smaller condition number than ∇ 2 a 1 a 1 L ( a 1 , a 2 ) , which corresponds to the problem that AM solves.
Proposition 2.
Given F deﬁned in (9), we have λ max
[Equation]
2 F ( a 1 ))
[Equation]
and
[Equation]
The proof is deferred to Appendix B.1.
As suggested by Proposition 2, F ( a 1 ) is much betterconditioned than L ( a 1 , a 2 ) in terms of a
2. S OLVING R WOC BY H YPERGRADIENT D ESCENT
We present how to solve (6) using our hypergradient approach.
Speciﬁcally, we compute the “hypergradient” of F (cid:15) ( w ) based on the following theorem.
Theorem 2.
The gradient of F (cid:15) with respect to w is
[Equation]
The deﬁnition of P and Q and the proof is deferred to Appendix C. Theorem 2 suggests that we ﬁrst solve the lower level problem in (6),
[Equation]
and then substitute S ∗ (cid:15) into (10) to obtain ∇ w F (cid:15) ( w ) .
The only question is to solve (11). This can be done efﬁciently using a variant of the Sinkhorn algorithm (Cuturi, 2013; Benamou et al., 2015).
Speciﬁcally, (11) can be formulated as an entropic optimal transport (EOT) problem (Monge, 1781; Kantorovich, 1960), which aims to ﬁnd the optimal way to transport the mass from a categorical distribution with weight µ = [ µ 1 , . . .
, µ n ] (cid:62) to another categorical distribution with weight ν
[Equation] [Equation]
, ν m ] (cid:62) ,
[Equation]
+
[Equation]
Γ) , with Π( µ, ν ) = { Γ ∈ R n × m : Γ 1 m = µ, Γ (cid:62) 1 n = ν, Γ ij ≥ 0 } , (12) where M ∈ R n × m is the cost matrix with M ij the transport cost.
When we set the proper cost matrix and the two categorical distributions as the empirical distributions of D 1 and D 2 , i.e.,
M = C ( w ) and µ = ν = 1 n /n, (12) becomes a scaled version of the lower problem of (6), and the optimal solution satisﬁes S ∗ (cid:15) =
n Γ ∗ .
Therefore, to obtain S ∗ (cid:15) in (12), we ﬁrst apply the Sinkhorn algorithm to solve the EOT problem in (12): At the (cid:96) -th iteration, we take p ( (cid:96) +1)
= µ
Gq ( (cid:96) ) and q ( (cid:96) +1) =
ν
G (cid:62) p ( (cid:96) +1) , where q (0) = 1 n 1 n and G ij = exp (cid:18) − C ij ( w ) (cid:15) (cid:19) ,
G ∈ R n × n , and the division here is entrywise.
Let p ∗ and q ∗ denote the stationary points.
Then we obtain S ∗ (cid:15),ij = np ∗ i G ij q ∗ j .
Remark 3.
The Sinkhorn algorithm is iterative and cannot exactly solve (11) in ﬁnite steps.
As the Sinkhorn algorithm is very efﬁcient and attains linear convergence, it sufﬁces to well-approximate the gradient ∇ w F (cid:15) ( w ) using the output inexact solution.
ROBOT FOR R OBUST C ORRESPONDENCE
We next propose a robust version of ROBOT to solve rRWOC (Varol & Nejatbakhsh, 2019).
Note that in (6), the constraint S ∈ Π( 1 n , 1 n ) enforces a one-to-one matching between D 1 and D 2 .
For rRWOC, however, such an exact matching may not exist.
For example, when n < m , where n = |D 1 | , m = |D 2 | .
Therefore, we need to relax the constraint on S .
Motivated by the connection between (6) and (12), we propose to solve the lower problem 3 :
[Equation]
13) subject to
[Equation]
where S ∗ r ( w ) ∈ R n × m denotes an inexact correspondence between D 1 and D 2 .
As can be seen in (13), we relax the marginal constraint Π( 1 , 1 ) in (6) to Π(¯ µ, ¯ ν ) , where ¯ µ, ¯ ν are required to not deviate much from 1 4 .
Illustrative examples of the exact and robust alignments are provided in Figure 1.
Computationally, (13) can be solved by taking the Sinkhorn iteration and the projected gradient iteration in an alternating manner (See more details in Appendix D).
Given S ∗ r ( w ) , we solve the upper level optimization in (6) to obtain w ∗ , i.e., w ∗ =
arg min w (cid:104) C ( w ) , S ∗ r ( w ) (cid:105) .
Similar to the previous section, we use a ﬁrst-order algorithm to solve this problem, and we derive explicit expressions for the update rules.
See Appendix E for details.
[Figure]
E XPERIMENT
[Figure]
We evaluate ROBOT and ROBOTrobust on both synthetic and realworld datasets, including ﬂow cytometry and multi-object tracking.
We ﬁrst present numerical results and then we provide insights in the discussion section.
Experiment details and auxiliary results can be found in Appendix G.
4. U NLABELED S ENSING
Data Generation .
We follow the unlabeled sensing setting (Tsakiris & Peng, 2019) and generate n = 1000 data points { ( y i , z i ) } ni =1 , where z i ∈ R e .
Note here we take d = 0 .
We ﬁrst generate z
i , w ∼ N ( 0 e , I e ) , and ε i ∼ N (0 , ρ 2noise ) .
Then we compute y i =
z (cid:62) i w + ε i .
We randomly permute the order of 50% of z i so that we lose the Z -to- Y correspondence.
We generate the test set in the same way, only without permutation.
Baselines and Training .
We consider the following scalable methods:
Oracle : Standard linear regression where no data are permuted.
Least Squares ( LS ): Standard linear regression, i
e., treating the data as if they are not permuted.
Alternating Minimization ( AM , Abid et al.
We iteratively solve the correspondence given w , and update w using gradient descent with the correspondence.
Stochastic EM (Abid & Zou, 2018):
A stochastic EM approach to recover the permutation.
Robust Regression ( RR , Slawski & Ben-David (2019); Slawski et al. (2019a)).
A two-stage block coordinate descent approach to discard outliers and ﬁt regression models.
Random Sample ( RS , Varol & Nejatbakhsh (2019)):
A random sample consensus (RANSAC) approach to estimate w .
We initialize AM, EM and ROBOT using the output of RS with multi-start.
We adopt a linear model f
( Z ; w ) =
Z (cid:62) w .
Models are evaluated by the relative error on the test set
, i.e., error = (cid:80) i
( (cid:98) y
i − y i ) 2 / (cid:80) i
( y i − ¯ y ) 2 , where (cid:98)
y i is the predicted label, and ¯ y is the mean of { y i } .
Results .
We visualize the results in Figure 2.
In all the experiments, ROBOT achieves better results than the baselines.
Note that the relative error is larger for all methods except Oracle as the dimension and the noise increase.
For low dimensional data, e.g., e = 5 , our model achieves even better performance than Oracle.
We have more discussions on using RS as initializations in Appendix G.5.
4. N ONLINEAR R EGRESSION
Data Generation .
We mimic the scenario where the dataset is collected from different platforms.
Speciﬁcally, we generate n data points { ( y i , [ x i , z i ]) } ni =1 , where x i ∈ R d and z i ∈ R e .
We ﬁrst generate x i ∼ N ( 0 d , I d ) , z i ∼ N ( 0 e , I e ) , w ∼ N ( 0 d + e , I d + e ) , and ε i ∼ N (0 , ρ 2noise ) .
Then we
[Figure]
Baselines and Training .
We consider a nonlinear function f
( X, Z ; w ) = (cid:80) dk
=1 sin ([ X, Z ] k w k ) .
In this case, we consider only two baselines — Oracle and LS, since the other baselines in the previous section are designed for linear models.
We evaluate the regression models by the transport cost divided by (cid:80) i ( y i − ¯ y ) 2 on the test set.
Results .
As shown in Figure 3, ROBOT-robust consistently outperforms ROBOT and LS, demonstrating the effectiveness of our robust formulation.
Moreover, ROBOT-robust achieves better performance than Oracle when the number of training data is large or when the noise level is high.
4. F LOW C YTOMETRY
[Figure]
In ﬂow cytometry (FC), a sample containing particles is suspended in a ﬂuid and injected into the ﬂow cytometer, but the measuring instruments are unable to preserve the correspondence between the particles and the measurements.
Different from FC, gated ﬂow cytometry (GFC) uses “gates” to sort the particles into one of many bins, which provides partial ordering information since the measurements are provided individually for each bin.
In practice, there are usually 3 or 4 bins.
Settings .
We adopt the dataset from Knight et al.
Following Abid et al.
(2017), the outputs
y i ’s are normalized, and we select the top 20 signiﬁcant features by a linear regression on the top 1400 items in the dataset.
We use 90% of the data as the training data, and the remaining as test data.
For ordinary
FC, we randomly shufﬂe all the labels in the training set.
For GFC, the training set is ﬁrst sorted by the labels, and then divided into equal-sized groups, mimicking the sorting by gates process.
The labels in each group are then randomly shufﬂed.
To simulate gating error, 1% of the data are shufﬂed across the groups.
We compare ROBOT with Oracle, LS, Hard EM (a variant of Stochastic EM proposed in Abid & Zou (2018)), Stochastic EM, and AM.
We use relative error on the test set as the evaluation metric.
Results .
As shown in Figure 4, while AM achieves good performance on GFC when the number of groups is 3, it behaves poorly on the FC task.
ROBOT, on the other hand, is efﬁcient on both tasks.
4. M ULTI -O BJECT T RACKING
In this section we extend our method to vision-based Multi-Object Tracking (MOT), a task with broad applications in mobile robotics and autonomous driving, to show the potential of applying
RWOC to more real-world tasks.
Given a video and the current frame, the goal of MOT is to predict the locations of the objects in the next frame.
Speciﬁcally, object detectors (Felzenszwalb et al., 2009; Ren et al., 2015)
ﬁrst provide us the potential locations of the objects by their bounding boxes.
Then, MOT aims to assign the bounding boxes to trajectories that describe the path of individual objects over time.
Here, we formulate the current frame and the objects’ locations in the current frame as D 2 = { z j } , while we treat the next frame and the locations in the next frame as
D 1 = { ( x i , y i ) } .
[Table]
Existing deep learning based MOT algorithms require large amounts of annotated data, i.e., the ground truth of the correspondence, during training.
Different from them, our algorithm does not require the correspondence between D 1 and D 2 , and all we need is the video.
This task is referred to as unsupervised MOT (He et al., 2019).
Related Works .
To the best of our knowledge, the only method that accomplishes unsupervised end-toend learning of MOT is He et al.
However, it targets tracking with low densities, e.g., Sprites-MOT , which is different from our focus.
Settings .
We adopt the MOT17 (Milan et al., 2016) and the MOT20 (Dendorfer et al., 2020) datasets.
Scene densities of the two datasets are 31 .
8 and 170 . 9 , respectively, which means the scenes are pretty crowded as we illustrated in Figure 5.
We adopt the DPM detector (Felzenszwalb et al., 2009) on MOT17 and the Faster-RCNN detector (Ren et al., 2015) on MOT20 to provide us the bounding boxes.
Inspired by Xu et al.
(2019b), the cost matrix is computed as the average of the Euclidean center-point distance and the Jaccard distance between the bounding boxes,
[Equation]
where c ( · ) is the location of the box center, H and W are the height and the width of the video frame, and J ( · , · ) is the Jaccard distance deﬁned as 1 -IoU (Intersection-over-Union).
We utilize the single-object tracking model SiamRPN 5 (Li et al., 2018) as our regression model f .
We apply
ROBOT-robust with ρ 1 =
ρ 2 = 10 − 3 .
See Appendix G for more detailed settings.
Results .
We demonstrate the experiment results in Table 1, where the evaluation metrics follow
Ristani et al.
In the table, ↑ represents the higher the better, and ↓ represents the lower the better.
ROBOT signiﬁes the model trained by ROBOT-robust, and w/o ROBOT means the pretrained model in Li et al.
The scores are improved signiﬁcantly after training with ROBOT-robust.
We also include the scores of the SORT model (Bewley et al., 2016) obtained from the dataset platform.
Different from SiamRPN and SiamRPN+ROBOT, SORT is a supervised learning model.
As shown, our unsupervised training framework achieves comparable or even better performance.
D ISCUSSION
Sensitivity to initialization .
As stated in Pananjady et al.
(2017b), obtaining the global optima of (1) is in general an NP-hard problem.
Some “global” methods use global optimization techniques and have exponential complexity, e.g., Elhami et al.
(2017), which is not applicable to large data.
Other “local” methods only guarantee convergence to local optima, and the convergence is very sensitive to initialization.
Compared with existing “local” methods, our method is computationally efﬁcient and greatly reduces the sensitivity to initialization.
[Figure]
To demonstrate such an advantage, we run AM and
ROBOT with 10 different initializations, and then we sort the results based on (a) the averaged residual on the training set, and (b) the relative prediction error on the test set.
We plot the percentiles in Figure 6.
Here we use fully shufﬂed data under the unlabeled sensing setting, and we set n = 1000 , e = 5 , ρ 2noise = 0 .
1 , and (cid:15) =
10 − 2 .
We can see that ROBOT can ﬁnd “good” solutions in 30% of the cases (The relative prediction error is smaller than 1 ), but AM is more sensitive to the initialization and cannot ﬁnd “good” solutions.
[Figure] [Figure]
[Figure]
ROBOT v.s. Automatic Differentiation (AD).
Our algorithm computes the Jacobian matrix directly based on the KKT condition of the lower problem (11).
An alternative approach to approximate the Jacobian is to use automatic differentiation to differentiate through the Sinkhorn iterations when solving (11).
As suggested by Figure 7 (a), running Sinkhorn iterations until convergence ( 200 Sinkhorn iterations) can lead to a better solution 6 .
In order to apply AD, we need to store all the intermediate updates of all the Sinkhorn iterations.
This means the memory usage is proportional to the number of iterations, which is not necessarily affordable.
In contrast, applying our explicit expression for the backward pass is memory efﬁcient.
Moreover, we also observe that AD is much more time-consuming than our method.
The time and memory comparisons are shown in Figure 7 (b)(c), where we set n = 1000 .
Connection to EM.
Abid & Zou (2018) adopt an Expectation Maximization (EM) method for RWOC, where S is modeled as a latent random variable.
In the M-step, we maximize the expected likelihood of the data over S .
This method shares the same spirit as ours: We avoid updating w using one single permutation matrix like AM.
However, this method is very dependent on a good initialization.
Speciﬁcally, if we randomly initialize w , the posterior distribution of S in this iteration would be close to its prior, which is a uniform distribution.
In this way, the follow-up update for w is not informative.
Therefore, the solution of EM would quickly converge to an undesired stationary point.
Figure 8 illustrates an example of converged correspondence, where we adopt n = 30 , o =
e = 1 , d = 0 .
For this reason, we initialize EM with good initial points, either by RS or AM throughout all experiments.
Related works with additional constraints.
There is another line of research which improves the computational efﬁciency by solving variants of RWOC with additional constraints.
Speciﬁcally,
Haghighatshoar & Caire (2017); Rigollet & Weed (2018) assume an isotonic function (note that such an assumption may not hold in practice), and Shi et al.
(2018); Slawski & Ben-David (2019); Slawski et al.
(2019a;b); Varol & Nejatbakhsh (2019) assume only a small fraction of the correspondence is missing.
Our method is also applicable to these problems, as long as the additional constraints can be adapted to the implicit differentiation.
More applications of RWOC.
RWOC problems generally appear for two reasons.
First, the measuring instruments are unable to preserve the correspondence.
In addition to GFC and MOT, we list a few more examples: SLAM tracking (Thrun, 2007), archaeological measurements (Robinson, 1951), large sensor networks (Keller et al., 2009), pose and correspondence estimation (David et al., 2004), and the genome assembly problem from shotgun reads (Huang & Madan, 1999).
Second, the data correspondence is masked for privacy reasons.
For example, we want to build a recommender system for a new platform, borrowing user data from a mature platform.
A CKNOWLEDGEMENT
This works is partially supported by NSF IIS-2008334.
Hongteng Xu was supported in part by
Beijing Outstanding Young Scientist Program (NO. BJJWZYJH012019100020098) and National
Natural Science Foundation of China (No. 61832017). Hongyuan Zha is supported in part by a grant from Shenzhen Institute of Artiﬁcial Intelligence and Robotics for Society. We also appreciate the fruitful discussions with Bo Dai and Yan Li A C ONNECTION BETWEEN OT AND RWOC
Theorem 1.
Denote Π( a, b ) = { S ∈ R n × m : S 1 m = a, S (cid:62) 1 n = b, S ij ≥ 0 } for any a ∈ R n and b ∈ R m .
Then at least one of the optimal solutions of the following problem lies in P .
[Equation]
Proof.
Denote the optimal solution of (14) as Z ∗ .
As we mentioned earlier, this is a direct corollary of Birkhoff–von Neumann theorem (Birkhoff, 1946; Von Neumann, 1953).
Speciﬁcally,
Birkhoff–von Neumann theorem claims that the polytope Π( 1 n , 1 n ) is the convex hull of the set of n × n permutation matrices, and furthermore that the vertices of Π( 1 n , 1 n ) are precisely the permutation matrices.
On the other hand, (14) is a linear optimization problem.
There would be at least one optimal solutions lies at the vertices given the problem is feasible.
As a result, there would be at least one
Z ∗ being a permutation matrix.
B T WO P ERSPECTIVES OF THE M OTIVATIONS OF B ILEVEL O PTIMIZATION B. F ASTER C ONVERGENCE
The bilevel optimization formulation has a better gradient descent iteration complexity than alternating minimization.
To see this, consider a quadratic function F ( a 1 , a 2 )
= a (cid:62) Pa + b (cid:62) a , where a 1 ∈ R d 1 , a 2 ∈ R d 2 , a =
[ a (cid:62) 1 , a (cid:62) 2 ] (cid:62)
∈ R ( d 1 + d 2 ) , P ∈ R ( d 1 + d 2 ) ×
( d 1 + d 2 ) , b ∈ R ( d
1 + d 2 ) .
To further simplify the discussion, we assume P =
ρ 1 ( d 1 + d 2 ) 1 (cid:62) ( d 1 + d 2 )
+ (1 − ρ )
I d
1 + d 2 , where I d 1 + d 2 is the identity matrix.
Then we have the following proposition.
Proposition 1.
Given F deﬁned in (9), we have
[Equation]
and λ [Equation] ∇ [Equation]
L
[Equation]
a
[Equation]
a
[Equation]
Proof.
For alternating minimization, the Hessian for a 1 is a submatrix of P , i.e.,
[Equation]
whose condition number is
[Equation]
We now compute the condition number for ROBOT.
Denote
P = (cid:20) P
P
P
P(cid:21) , b = [Equation] bb(cid:21) , where P 11 ∈ R d 1 × d 1 , P 12 ∈ R d 1 × d 2 , P 21 ∈ R d 2 × d 1 , P 22 ∈ R d 2 × d 2 , and b 1 ∈ R d 1 , b 2 ∈ R d 2 .
ROBOT ﬁrst minimize over a 2 ,
[Equation]
Substituting a ∗ 2 ( a 1 ) into F ( a 1 , a 2 ) , we can obtain the Hessian for a 1 is
[Equation]
Using Sherman–Morrison formula, we can explicitly express P − 1 22 as
[Equation]
Substituting it into H ROBOT ,
[Equation]
Therefore, the condition number is
[Equation]
Note that C AM increases linearly with respect to d 1 .
Therefore, the optimization problem inevitably becomes ill-conditioned as dimension increase.
In contrast, C ROBOT can stay in the same order of magnitude when d 1 and d 2 increase simultaneously.
Since the iteration complexity of gradient descent is proportional to the condition number (Bottou et al., 2018), ROBOT needs fewer iterations to converge than AM.
C D IFFERENTIABILITY
Theorem 2.
For any (cid:15) > 0 , S ∗ (cid:15) ( w ) is differentiable, as long as the cost C ( w ) is differentiable with respect to w .
As a result, the objective L (cid:15) ( w ) = (cid:104) C ( w ) , S ∗ (cid:15) ( w ) (cid:105) is also differentiable.
Proof.
The proof is analogous to Xie et al.
We ﬁrst prove the differentiability of S ∗ (cid:15) ( w ) .
This part of proof mirrors the proof in Luise et al.
By Sinkhorn’s scaling theorem (Sinkhorn & Knopp, 1967),
[Equation]
Therefore, since C ij ( w ) is differentiable, Γ ∗ ,(cid:15) is differentiable if
[Equation]
w
[Equation]
, ζ ∗
[Equation]
w )) is differentiable as a function of w .
Let us set
L (
[Equation]
[Equation]
[Equation]
[Equation]
[Equation] = [Equation] and recall that ( ξ ∗ , ζ ∗ ) = arg max ξ,ζ L ( ξ, ζ ; µ, ν, C ) .
The differentiability of ( ξ ∗ , ζ ∗ ) is proved using the Implicit Function theorem and follows from the differentiability and strict convexity in ( ξ ∗ , ζ ∗ ) of the function L .
Theorem 3.
Denoting L (cid:15) =
(cid:104) C ( w ) , S ∗
[Equation]
w )
[Equation]
The gradient of L (cid:15) with respect to w is
[Equation]
with
[Equation]
[Table] ∗
[Table]
µ )) −
[Table]
¯
[Table]
∗ (cid:15) , ¯ ν = ν 1: n − 1 , ¯ S ∗ (cid:15) =
S ∗ (cid:15), 1: n, 1: n − 1 .
Proof.
This result is straightforward combining the Sinkhorn’s scaling theorem and Theorem 3 in
Xie et al.
D A LGORITHM OF THE F ORWARD P ASS FOR ROBOT- ROBUST
For better numerical stability, in practice we add two more regularization terms,
[Equation]
where h (¯ µ ) = (cid:80) i ¯ µ i log ¯ µ i is the entropy function for vectors.
This can avoid the entries of ¯ µ and ¯ ν shrink to zeros when updated by gradient descent.
We remark that since we have entropy term H ( S ) , the entries of S would not be exactly zeros.
Furthermore, we have ¯ µ = S 1 and ¯ µ = S 1 .
Therefore, theoretically the entries of ¯ µ and ¯ ν will not be zeros.
We only add the two more entropy terms for numerical consideration.
The detailed algorithm is in Algorithm 1.
Although the algorithm is not guaranteed to converge to a feasible solution, in practice it usually converges to a good solution (Wang et al., 2015). Algorithm
Require: C ∈ R m × n , µ, ν, K, (cid:15), L, η
G ij = e − Cij(cid:15) ¯ µ = µ, ¯ ν = ν b = 1 n for l = 1 , · · · , L do a = ¯ µ/ ( Gb ) , b = ¯ ν/ ( G T a ) ¯ µ = ¯ µ − η
( e a(cid:15) + (cid:15) 1 ∗ log ¯ µ ) , ¯ ν =
¯ ν − η ( e b(cid:15) + (cid:15) 2 ∗ log ¯ ν ) ¯ µ = max { ¯ µ, 0 } , ¯ ν = max { ¯ ν, 0 } ¯ µ = ¯ µ/ (¯ µ (cid:62) 1 ) ,
¯ ν = ¯ ν/ (¯ ν (cid:62) 1 ) if (cid:107) ¯ µ − µ (cid:107) 22 > ρ 1 then ¯ µ = µ + √ ρ 1 ¯ µ − µ (cid:107) ¯ µ − µ (cid:107)end if if (cid:107) ¯ ν − ν (cid:107) 22 >
ρ 2 then ¯ ν = ν
+ √ ρ 2 ¯ ν − ν (cid:107) ¯ ν − ν (cid:107)end if end for
S = diag( a ) (cid:12) G (cid:12) diag( b )
E A LGORITHM OF THE B ACKWARD P ASS FOR ROBOT- ROBUST
Since the derivation is tedious, we ﬁrst summarize the outline of the derivation, then provide the detailed derivation.
E. S UMMARY
Given ¯ µ ∗ , ¯ ν ∗ , S ∗ r ( w ) , we compute the Jacobian matrix dS ∗ r ( w ) /dw
using implicit differentiation and differentiable programming techinques.
Speciﬁcally, the Lagrangian function of Problem (16) is
[Equation]
where ξ and ζ are dual variables.
The KKT conditions (Stationarity condition) imply that the optimal solution Γ ∗ ,(cid:15) can be formulated using the optimal dual variables ξ ∗ and ζ ∗ as,
[Equation]
By the chain rule, we have
[Equation]
Therefore, we can compute dS ∗ r ( w ) /dw
if we obtain dξ ∗ dC and dζ ∗ dC .
Substituting (17) into the Lagrangian function, at the optimal solutions we obtain
[Equation]
Denote r ∗ =
[( ξ ∗ ) (cid:62) , ( ζ ∗ ) (cid:62) , (¯ µ ) (cid:62) , (¯ ν ) (cid:62) , λ ∗ 1 , λ ∗ 2 , λ ∗ 3 , λ ∗ 4 ] (cid:62) , and φ ( r ∗ ; C ) =
∂
L ( r ∗ ; C ) /∂r ∗ .
At the optimal dual variable r ∗ , the KKT condition immediately yields φ ( r ∗ ; C ) ≡ 0 .
By the chain rule, we have
[Equation]
Rerranging terms, we obtain
[Equation]
Combining (17), (18), and (19), we can then obtain dS ∗ r ( w ) /dw .
E. D ETAILS
Now we provide the detailed derivation for computing dS ∗ r /dw .
Since S ∗ r is the optimal solution of an optimization problem
[Equation]
we can follow the implicit function theorem to solve for the closed-form expression of the gradient.
Speciﬁcally
[Equation]
we adopt F (¯ µ, ν ) =
(cid:80) i (¯ µ i − µ i ) 2 , and rewrite the optimization problem as
[Equation]
The Language of the above problem is
[Equation]
j ¯
[Equation] − [Equation] ( (cid:88) i (¯ µ
[Equation]
µ
[Equation] λ [Equation] ( (cid:88) j (¯ ν j − ν j )
[Equation] − ρ 2 ) .
Easy to see that the Slater’s condition holds.
Denote
L ∗ =
L ( C, S ∗ r , ¯ µ ∗ , ¯ ν ∗ , ξ ∗ , ζ ∗ , λ ∗ 1 , λ ∗ 2 , λ ∗ 3 , λ ∗ 4 ) .
Following the KKT conditions,
[Equation]
Therefore, S ∗
[Equation] [Equation]
=
[Equation]
Then we have
[Equation]
So all we need to do is to compute dξ ∗ dC and dζ ∗ dC .
Denote F ij = e ξi + ζj − Cij (cid:15) .
Denote
[Equation]
Denote χ =
[ χ 1 , χ 2 , χ 3 , χ 4 ] , and λ =
[ λ 1 , λ 2 , λ 3 , λ 4 ] .
Following the KKT conditions, we have
[Equation]
at the optimal solutions.
Therefore, for the optimal solutions we have [Equation]
[Equation] dC
[Equation] dC [Equation]
dC [Equation] 0 , dp [Equation]
dξ
[Equation]
dζ [Equation] dC [Equation]
d [Equation] dC
[Equation] 0 , dq dC
[Equation] dC [Equation]
ν [Equation]
dC [Equation] dχ [Equation]
dζ [Equation] dC
[Equation]
d [Equation] dC
[Equation] ∂χ [Equation] d
[Equation] dC [Equation] ∂χ
[Equation] dλ [Equation] dC = 0 .
Therefore, we have
[Equation]
 = −  
 ∂φ ∂ξ ∗ ∂φ ∂ζ ∗ ∂φ ∂ ¯ µ ∗ ∂φ ∂ ¯ ν ∗ ∂φ ∂λ ∗ ∂ψ ∂ξ ∗ ∂ψ ∂ζ ∗ ∂ψ ∂ ¯ µ ∗ ∂ψ ∂ ¯ ν ∗ ∂ψ ∂λ ∗ ∂p ∂ξ ∗ ∂p ∂ζ ∗ ∂p ∂ ¯ µ ∗ ∂p ∂ ¯ ν ∗ ∂p ∂λ
∗ ∂q ∂ξ ∗ ∂q ∂ζ ∗
∂q ∂ ¯ µ ∗ ∂q
∂ ¯ ν ∗
∂q ∂λ ∗ ∂χ ∂ξ ∗ ∂χ ∂ζ ∗ ∂χ ∂ ¯ µ ∗ ∂χ ∂
¯ ν ∗ ∂χ ∂λ ∗ 
  − 1  
 ∂φ ∂C∂ψ ∂C∂p ∂C∂q ∂C∂χ ∂C 
  .
After some derivation, we have   dξ ∗ dCdζ ∗ dCd ¯ µ ∗ dCd ¯ ν ∗ dCdλ ∗dCdλ ∗dCdλ ∗dCdλ ∗dC 

[Figure]
0 2 λ 3 (¯ µ − µ ) (cid:62)
(cid:107) ¯ µ − µ (cid:107) 22 − ρ0  2 λ 4 (¯ ν − ν ) (cid:62)   (cid:107) ¯
ν
− ν (cid:107) 22 − ρ  − 1   ∂φ ∂C∂ψ ∂C 
 , and ∂φ h
[Equation]
To efﬁciently solve for the inverse in the above equations, we denote
[Equation]

[Equation]
[Equation]
(cid:62)   ,
[Equation]
We ﬁrst A − 1 using the rules for inverting a block matrix,
[Equation]
where
[Equation]
Then using the rules of inverting a block matrix again, we have
[Equation]
Therefore, the bottleneck of computation is the inverting step in computing K .
Note L is a diagonal matrix, we can further lower the computation cost by applying the rules for inverting a block matrix again.
The value of λ 3 and λ 4 can be estimated from the fact p = 0 , q = 0 .
We detail the algorithm in Algorithm 2.
Algorithm
Require: C ∈ R m × n , µ, ν, (cid:15), dCdw
Run forward pass to get S
= S ∗ r , ¯ µ, ¯ ν, ξ, ζ x 1 = (cid:80)
[Table]
[Table]
[Table]
¯ µ = ¯ µ + (cid:15) (2 λ 3 1 n + (cid:15)
[Table]
¯
[Table]
) − 1 , ¯ ν = ¯ ν + (cid:15) (2 λ 4 1 m + (cid:15)
[Table]
[Table]
¯ ν (cid:48) = ¯ ν [: − 1] , S (cid:48) =
S
[: , : − 1]
K ← diag(¯ ν (cid:48)
[Table]
− ( S [Table] T (diag [Table]
S
[Table]
H [Table] − 1 + (diag(¯ µ )) − 1 S (cid:48)
K − 1 ( S (cid:48) ) (cid:62) (diag(¯ µ )) −
H 2 ← − (diag(¯ µ )) − 1 S (cid:48)
K −
H 3 ← ( H 2 ) (cid:62)
H 4 ← K −
Pad H 2 to be [ n, m ] with value
Pad H 3 to be [ m, n ] with value
Pad H 4 to be [ m, m ] with value
L = diag([ (cid:15) (2 λ 3 1 n + (cid:15) 1 ¯ µ ) − 1 , (cid:15) (2 λ 4 1 m + (cid:15) 2 ¯ ν ) − 1 ])
A 1 =
[ H 1 , H 2 ; H 3 , H 4 ]
A 2 = − A 1 · L
A 3 = A (cid:62)
A 4 = L + L · A 1 · L
E =
A 1 + A 2 · B 1( D − C · A 4 · B ) − 1 C · A 3 , where B 1 , C 1 , D deﬁned above [ J 1 , J 2 ; J 3 , J 4 ] =
E ,
where J 1 ∈ R n × n , J 2 ∈ R n × m , J 3 ∈ R m × n , J 4 ∈ R m × m [ dξ ∗ dC ]
nij ← [ J 1 ] ni S ij + [ J 2 ] nj S ij [ dζ ∗ dC ]
mij ← [ J 3 ] mi S ij + [ J 4 ] mj S ij
Pad dζ ∗ dC to be [ m, n, m ] with value[ d L dC ] ij ← 1 (cid:15) ( − C ij S ij + (cid:80) n,m C nm S
nm [ da ∗ dC ] nij + (cid:80) n,m C nm S nm [ db ∗ dC ] mij )
+ S
ij return
d
L dC dC dw
[Figure]
F D IFFERENT FORMS OF MARGINAL RELAXATION
In this paper we adopt F to be the Euclidean distance.
This is because this choice provides an
OT plan that ﬁts our intuition – the data points with signiﬁcantly larger transportation cost should not be considered.
Figure 9 shows an illustration.
Here, the input distributions are the empirical distributions of the scalars on the left and the bottom.
Notice that there are three support points in µ that are far away from others, i.e., 10 . 72 , 10 . 89 , 10 . 96 .
In Figure 9 (a), the optimal solution Γ ∗ r automatically ignores them, matching only the rest of the scalars.
One alternative choice of F is the
Kullback–Leibler (KL) divergence (Chizat et al., 2018b), whose resulted formulation possesses an efﬁcient algorithm for the forward pass, and the differentiability for the backward pass.
We do not adopt it because the OT plan generated by this choice does not ﬁt out intuition: As shown in Figure 9 (b), the OT plan tends to ignore the points that are away from the mean, even with a very small ρand ρ 2 .
For both ﬁgures, we adopt (cid:15) =
10 − 5 .
G M ORE ON E XPERIMENTS G. U NLABELED S ENSING
We now provide more training details for experiments in Section 4.1.
Here, AM and ROBOT is trained with batch size 500 and learning rate 10 − 4 for 2 , 000 iterations.
For the Sinkhorn algorithm in ROBOT we set (cid:15) = 10 − 4 .
We run RS for 2 × 10 5 iterations with inlier threshold as 10 − 2 .
Other settings for the hyper-parameters in the baselines follows the default settings of their corresponding papers.
G. N ONLINEAR R EGRESSION
For the nonlinear regression experiment in Section 4.2, ROBOT and ROBOT-robust is trained with learning rate 10 − 4 for 80 iterations.
For n = 100 , 200 , 500 , 1000 , 2000 , we set batch size 10 , 30 , 50 , 100 , 300 , respectively.
We set (cid:15) =
10 − 4 for the Sinkhorn algorithm in ROBOT.
For Oracle and LS, we perform ordinary regression model and ensure convergence, i.e., learning rate 5 × 10 −for 100 iterations.
G. F LOW C YTOMETRY
We provide more details for the Flow Cytometry experiment in Section 4.3.
In the FC seting,
ROBOT is trained with batch size 1260 and learning rate 10 − 4 for 80 iterations.
In the GFC seting,
ROBOT is trained with batch size 1260 and learning rate 6 × 10 − 4 for 60 iterations.
We set (cid:15) =
10 −for the Sinkhorn algorithm in ROBOT.
Other settings for the hyper-parameters in the baselines follows the default settings of their corresponding papers.
EM is initialized by AM.
G. M ULTI -O BJECT T RACKING
For the MOT experiments in Section 4.4, the reported results of MOT17 (train) and MOT17 (dev) is trained on MOT17 (train), and the reported results of MOT20 (train) and MOT20 (dev) is trained on MOT20 (train).
Each model is trained for 1 epoch.
We adopt Adam optimizer with learning rate = 10 − 5 , (cid:15) =
10 − 4 , and η = 10 − 3 .
To track the birth and death of the tracks, we adapt the inference code of Xu et al.
(2019b).
G. C OMBINATION WITH RS
[Table]
As suggested in Figure 2, although RS cannot perform well itself, retraining the output of RS using our algorithms increases the performance by a large margin.
To show that combining RS and ROBOT can achieve better results than RS alone, we compare the following two cases: i).
Subsample 2 × 10 5 times using
RS; ii).
Subsample 10 5 times using RS followed by ROBOT for 50 training steps.
The result is shown in Table 2.
For a larger permutation proportion, RS alone cannot perform as well as RS+ROBOT combination.
Here, we have 10 runs for each proportion.
We adopt SNR = 100 , d = 5 for data, and (cid:15) =
10 − 4 , learning rate 10 − 4 for ROBOT training.
G. T HE E FFECT OF ρ 1 AND ρ
We visualize S ∗ r computed from the robust optimal transport problem in Figure 10.
The two input distributions are Unif( 0 , 2 ) and Unif( 0 , 1 ).
We can see that with large enough ρ 1 and ρ 2 , Unif( 0 , 1 ) would be aligned with the ﬁrst half of Unif( 0 , 2 )
[Figure]
G.
C OMPARISON OF R ESIDUALS IN L INEAR R EGRESSION
Settings .
We generate n data points { ( y i , [ x i , z i ]) } ni =1 , where x i ∈ R d and z i ∈ R e .
We ﬁrst generate x i ∼ N ( 0 d , I d ) , z i ∼ N ( 0 e , I e ) , w ∼ N ( 0 d + e , I d + e ) [Figure] and ε i ∼ N (0 , ρ 2noise ) .
Then we compute y i =
f
([ x i , z i ]; w ) + ε i .
Next, we randomly permute the order of { z i } so that we lose the data correspondence.
Here, D 1 = { ( x i , y i ) } and D 2 = { z j } mimic two parts of data collected from two separate platforms.
We adopt a linear model f ( x ; w ) =
x (cid:62) w .
To evaluate model performance, we use error =
(cid:80) i ( (cid:98)
y i − y i ) 2 / (cid:80) i
( y i − ¯ y ) 2 , where (cid:98)
y i is the predicted label, and ¯ y is the mean of { y i } .
Baselines .
We use Oracle, LS, Stochastic-EM as the baselines.
Notice that without a proper initialization, Stochastic-EM performs well in partially permuted cases, but not in fully shufﬂed cases.
For better visualization, we only include this baseline in one experiment.
Furthermore, we adopt two new baselines: Sliced-GW (Vayer et al., 2019) and Sinkhorn-GW (Xu et al., 2019a), which can be used to align distributions and points sets.
Results .
We visualize the ﬁtting error of regression models in Figure 11.
We can see that ROBOT outperforms all the baselines except Oracle.
Also, our model can beat the Oracle model when the dimension is low or when the noise is large.