Keypoint-Guided Optimal Transport with Applications in Heterogeneous Domain Adaptation
Abstract
Existing Optimal Transport (OT) methods mainly derive the optimal transport plan/matching under the criterion of transport cost/distance minimization, which may cause incorrect matching in some cases.
In many applications, annotating a few matched keypoints across domains is reasonable or even effortless in annotation burden.
It is valuable to investigate how to leverage the annotated keypoints to guide the correct matching in OT.
In this paper, we propose a novel KeyPoint-Guided model by ReLation preservation (KPG-RL) that searches for the matching guided by the keypoints in OT.
To impose the keypoints in OT, first, we propose a maskbased constraint of the transport plan that preserves the matching of keypoint pairs.
Second, we propose to preserve the relation of each data point to the keypoints to guide the matching.
The proposed KPG-RL model can be solved by the Sinkhorn’s algorithm and is applicable even when distributions are supported in different spaces.
We further utilize the relation preservation constraint in the Kantorovich
Problem and Gromov-Wasserstein model to impose the guidance of keypoints in them.
Meanwhile, the proposed KPG-RL model is extended to partial OT setting.
As an application, we apply the proposed KPG-RL model to the heterogeneous domain adaptation.
Experiments verified the effectiveness of the KPG-RL model.
Code is available at https://github.com/XJTU-XGU/KPG-RL .
Introduction
As a mathematical tool for distribution alignment, mass transport, etc. , Optimal Transport (OT) [1] has gained increasing attention in the machine learning community.
OT aims to derive a transport map or plan between a source and a target distribution, such that the transport cost is minimized.
Due to its capacity to exploit the geometric property of data, OT has been employed in many applications, e .
g ., computer vision [2, 3], natural language processing [4, 5, 6, 7], generative adversarial network [8], domain adaptation [9], clustering [10], anomaly detection [11], etc .
OT has two typical formulations, i .
e ., the Monge’s formulation [12] and the Kantorovich’s formulation [13].
The Kantorovich’s formulation [13] relaxes the Monge’s formulation and attracts broader studies in applications.
Unless otherwise stated, by OT, we refer to the Kantorovich’s formulation in this paper.
The original OT model, i .
e ., the Kantorovich Problem [13] (KP), is a linear program that is computationally expensive.
The entropy-regularized OT [14] introduces the entropy of the transport plan as regularization to the
OT model which is solved by the computationally cheaper Sinkhorn-Knopp algorithm [15] allowing to use automatic differentiation [16].
KP needs to transport all the mass of source distribution to exactly match the mass of target distribution.
But in some cases, only partial mass of source and target distributions should be matched, e .
g ., the source or target samples contain outliers.
To overcome this limitation, the partial OT [17, 18, 19, 20],
[Figure]
unbalanced OT [21, 22], and robust OT [23, 24, 25] models are presented, that allow to transport only partial mass of distribution.
Another extension of KP is the Gromov-Wasserstein (GW) model [26, 27] that computes the distance between metrics defined within each domain rather than between samples across domains as in KP.
In most of the above models, the main criterion for the optimal transport plan/matching is by the minimization of total transport distance or distortion over all samples.
Though having achieved promising results in many applications, without any additional guidance, the OT models may lead to incorrect matching of samples.
Figures 1(b) and 1(c) illustrate examples of incorrect matching produced by KP and GW models.
In many applications, it is reasonable to annotate some paired keypoints across domains for guiding the matching in OT.
For instance, in non-rigid point set or image registration [28, 29], a few keypoint pairs that should be matched are annotated in two point sets/images.
In semi-supervised domain adaptation [30] and heterogeneous domain adaptation [31], there are a few labeled target domain data and large amount of labeled source domain data available, that could be directly taken as keypoints.
Therefore, it is valuable and important to investigate how to take advantage of those keypoints to guide the correct matching in OT.
Figure 1(d) shows an example that with the guidance of a few keypoints, the correctness of matching can be improved.
In this paper, we propose a novel KeyPoint-Guided model by ReLation preservation (KPG-RL) for leveraging the annotated keypoints to guide the matching in OT.
In KPG-RL, we first preserve the matching of keypoint pairs in OT using a mask-based constraint of the transport plan.
We then propose to preserve the relation of each data point to the keypoints in transport, enforcing the matching of the data points near paired keypoints across domains.
The proposed KPG-RL model is applicable even when distributions lie in different spaces, and can be solved by Sinkhorn’s algorithm.
We further enforce the relation preservation constraint in KP and GW to impose the guidance of keypoints in them.
To tackle the problems that only partial mass should be transported, we extend the KPG-RL model to partial OT setting, forming partial-KPG-RL model.
As an application, we apply the KPG-RL model to the heterogeneous domain adaptation (HDA) [31].
HDA is a transfer learning task that aims to transfer the knowledge of large-scale labeled source domain data to the target domain where a few labeled and larger amounts of unlabeled data are available for training.
The “heterogeneity” implies that the source and target domain data are in heterogeneous feature spaces, e .
g ., generated by different deep networks.
This heterogeneity poses a major obstacle in adapting the source trained model to the target domain.
We take the labeled target domain data and source class centers as keypoints and transport source domain data to target domain by our KPG-RL model.
Upon the transported source domain data and the labeled target domain data, a classification model is trained that is transferable to the target domain.
Experiments show that the
KPG-RL model is effective for HDA.
In the following sections, we discuss the related works in Sect.
2, and introduce the background of
OT in Sect.
In Sect.
4, we discuss the details of the proposed KPG-RL model.
In Sect.
5, we apply the KPG-RL model to HDA.
Section 6 concludes this paper.
Notations.
Σ m = { p ∈ R m + | (cid:80)
i p i = 1 } is the probability simplex.
⟨· , ·⟩ F is the Frobenius dot product of two matrices.
⊙ stands for the Hadamard product.
1 m denotes m -
Related Works
We review below the most related OT models and HDA methods to our work.
OT models.
The GW model [27] seeks a “distance-preserving” transport plan such that the distance between transported points in target domain is the same as the distance between the original points in source domain.
Our KPG-RL model aims to use keypoint pairs to guide the matching in OT by preserving the relation of each point to the keypoints.
Our “relation-preserving” scheme preserves the relation of data w.r.t.
the given keypoints, different from the pairwise distance-preserving constraint in GW.
We experimentally verified the effectiveness of relation-preserving scheme for introducing the guidance of keypoints in OT.
From the computational point of view, GW is a non-convex quadratic program, while KPG-RL is a linear program.
Lin et al .
[32] use the anchors to encourage clustering of data and to impose rank constraints on the transport plan to improve its robustness to outliers.
The “anchors” in [32] are intermediate points in computation for improving robustness, different from the “keypoints” in this paper, which are the annotated paired data for guiding the matching in OT.
Hierarchical OT [7, 33, 34] transports points by dividing them into some subgroups and then derives the transportation of these subgroups using OT.
Different in goal and methodology from Hierarchical
OT, we impose the guidance of keypoints for pursuing correct matching in OT by preserving the relation to the keypoints.
We do not explicitly divide the points into subgroups, and there is no hierarchy in our method.
TLB [27, 35] is a lower bound of GW that can be computed faster.
TLB takes the ordered distance of each point to all the points in the same domain as features, and then performs the Kantorovich formulation of OT using such features.
Differently, our method uses a carefully designed relation of each point to the keypoints to impose the guidance of keypoints to the other points.
Courty et al.
[9] constrain the cost function to encourage the matching of labeled data across source and target domains that share the same class labels for domain adaptation.
They use the Laplacian regularization to preserve the data structure.
Differently, we explicitly model the guidance of keypoints matching to the other data points in our OT formulation. The matching of paired keypoints is enforced by our mask-based constraint on the transport plan.
Zhang et al.
[36] propose Masked OT model as a regularization term to preserve the local feature invariances between finetuned and pretrained graph neural network (GNN) for the fine-tuning of GNNs.
Though the Masked OT [36] shares similar spirits to the mask-based modeling in our method, our main contribution is the relation preservation for imposing the guidance of keypoints, different from [36] in methodology.
For our mask-based modeling, it is utilized to impose the matching of keypoints which is theoretically guaranteed.
While the mask in [36] aims to preserve the local information of finetuned network from pretrained models.
The motivation and design of our mask are different from those in [36].
HDA methods.
HDA methods could be roughly categorized into cross-domain mapping and common subspace learning methods.
The cross-domain mapping approaches [31, 37, 38, 39, 40] learn a transform to map the source features or model parameters to target domain to achieve adaptation.
The common subspace learning approaches [41, 42, 43, 44, 45, 46] learn domain-specific projections to map source and target domain data into a common subspace such that their distributions are aligned.
Our method for HDA belongs to the first category, and could be mostly related to [37].
The method in [37] transports source samples to target domain using GW model regularized by the distance between the center of transported source samples and the center of labeled target samples having the same class labels.
Different from [37], we take each labeled target domain data and its corresponding source domain class center as a paired keypoint and preserve the relation of each data to the set of keypoints in each domain when conducting OT.
Background on Optimal Transport
Kantorovich Problem (KP).
We consider two sets of data points, i .
e ., the source data X
= { x i } mi = and the target data Y =
{ y j } nj =1 , of which the empirical distributions are p
=
(cid:80) mi =1 p i δ x i and q = (cid:80) nj =
1 q j δ y j .
With a slight abuse of notations, we may also denote p
=
( p 1 , p 2 , · · · , p m )
⊤ ∈
Σ m and q = ( q 1 , q 2 , · · · , q n ) ⊤ ∈ Σ n as the mass supported on X and Y respectively.
We define the cost matrix between X and Y as C =
( C i,j ) ∈
R m × n with C i,j = c ( x i , y j ) , where c is a cost function, which is set to the squared L 2 -distance of x i and y j in our experiments.
OT aims to optimally transport p towards q at the smallest cost, formulated as the following Kantorovich Problem (KP):
[Equation]
1 3 6 3 6𝒑𝒑 𝒒𝒒 𝑞𝑞 1 𝑞𝑞 2 𝑞𝑞 3 𝑞𝑞 4 𝑞𝑞 5 𝑞𝑞𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝     �𝜋𝜋0 0
0 0 0�𝜋𝜋 �𝜋𝜋 𝑀𝑀 𝜋𝜋 1 0 1 1 01 0 1 1 0  0 0 01 0 1 1 01 0 1 1 00 0 0 0 1𝒑𝒑 𝒒𝒒
When c is taken as a distance metric ( aka.
ground metric), the minimum value of objective function in Eq.
(1) is a distance between p and q , named Wasserstein distance.
Partial OT model.
The KP in Eq.
(1) takes the mass preserving assumption that all the mass of p is transported to exactly match the mass of q .
In many applications, only partial mass should be transported.
The partial OT model [18, 19] seeks the minimal cost of transporting only s unit mass from p to q , where 0 ⩽ s ⩽ min( ∥ p ∥ 1 , ∥ q ∥ 1 ) , formulated as
[Equation]
Note that in partial OT, the total mass ∥ p ∥ 1 and ∥ q ∥ 1 are not necessarily equal.
In the definition of Π( p , q ) and Π s ( p , q ) , m and n are respectively the length of p and q .
Gromov-Wasserstein (GW) model.
If the data points x i and y j lie in different spaces, the distance between x i and y j may not be computed.
The GW [27] model then minimizes the distortion when transporting the whole set of points from one space to another.
The GW model relies on the intradomain distance of source domain as C s =
( C si,k ) ∈
R m × m and target domain as C t =
( C tj,l ) ∈
R n × n where C si,k is the distance between source domain data x i and x k , and C tj,l is the distance between target domain data y j and y l .
The GW model is given by
[Equation]
Keypoint-Guided Optimal Transport
This section details our proposed keypoint-guided model that leverages the keypoints to guide the matching in OT.
The guidance is imposed by preserving the matching of keypoint pairs and the relation of each data point to the keypoints.
We first discuss the preservation of matching of keypoints, then introduce the modeling of relation, and finally present the keypoint-guided OT models.
Preservation of matching of keypoints in transport.
We denote the set of keypoint index pairs as K =
{ ( i u , j u ) }
Uu =1 with U denoting the number of paired keypoints.
We respectively denote
I =
{ i u } Uu =1 and J = { j u } Uu =1 as the sets of source and target keypoint indexes.
For the example illustrated in Fig.
2, K = { (3 , 2) , (6 , 5) } , I = { 3 , 6 } , and J = { 2 , 5 } .
To impose the guidance of these keypoints with indexes in K in deriving the matching in OT, we first guarantee the matching of the keypoint pairs.
As illustrated in Fig.
2, we preserve the matching of keypoints in transport using a mask-based constraint of the transport plan, which is motivated by the following observation.
If the paired keypoints ( i, j ) ∈ K are matched, the optimal transport plan ˜ π satisfies that the i -th row and j -th column of ˜ π must be zeros except ˜
π
i,j , which means that the all mass of source keypoint x i must be transported to target keypoint y j and y j can only receive the mass from x i .
For the example in
Fig.
2, the 3-th row and 2-th column are zeros except that ˜ π 3 , 2 > 0 .
This sparsity of ˜ π motivates us to model it as the Hadamard product of a mask matrix M =
( M i,j ) ∈
R m × n and a matrix π ∈ R m × n + with positive entries, i .
e .,
[Equation]
With Eq.
(4), we define the admissible solution set for our keypoint-guided OT model as
[Equation]
Note that the π in Eq.
(5) is not necessarily a coupling.
The entry M i,j of the mask matrix M is set to 0 if ˜ π
i,j needs to be 0, otherwise M i,j is set to 1.
Figure 2 illustrates the mask-based modeling of
Eq.
M is constructed as in Proposition 1.
Proposition 1.
Suppose that the mask matrix M satisfies that
[Equation]
and p i =
q j , for ( i, j ) ∈ K .
Then, the transport plan ˜ π
= M ⊙ π with π ∈ Π( p , q ; M ) preserves the matching of paired keypoints with index pairs in K .
More explanations of the construction of M and the proof of Proposition 1 are provided in Appendix A.1.
Proposition 1 indicates that if p i = q j , for ( i, j ) ∈ K , the matching of keypoint pairs is preserved by the mask-based constraint.
This mask-based modeling in Eq.
(4) is also applicable even for the case that there exist some ( i,
j ) ∈ K such that p i ̸ = q j .
For this case, we shall use different mask matrices.
Please refer to Appendix A.1 for the details.
Modeling the relation to keypoints.
To use the keypoints to guide the matching in transport, we propose to preserve the relation of each point to the set of keypoints in transport.
Figureillustrates the relation within points of each distribution.
For the data point x k ∈ X , its relation score
[Figure]
[Equation]
[Figure]
[Equation]
[Figure]
𝑗𝑗𝑗𝑗𝑗𝑗𝑘𝑘 𝑙𝑙 𝑅𝑅 𝑘𝑘 , 𝑖𝑖 1 𝑠𝑠 𝑅𝑅 𝑘𝑘 , 𝑖𝑖 3 𝑠𝑠 𝑅𝑅 𝑘𝑘 , 𝑖𝑖 2 𝑠𝑠 𝑅𝑅 𝑙𝑙 , 𝑗𝑗 3 𝑡𝑡 𝑅𝑅 𝑙𝑙 , 𝑗𝑗 2 𝑡𝑡 𝑅𝑅 𝑙𝑙 , 𝑗𝑗 1 𝑡𝑡
Source
Target to the keypoint x
i u for i u ∈
I (illustrated by red circle in the left of Fig. 3) is defined as
[Equation]
where C sk,i u is the l 2 -distance between x k and x i u , and τ is temperature set as τ = ρ ∗ max i,k { C si,k } in experiments.
Similarly, for y l ∈ Y , its relation score to the keypoint y j u for j u ∈ J is defined as
[Equation]
where C tl,j u is the distance between y l and y j u , τ ′ is temperature set to ρ ∗ max j,l { C tj,l } .
The setting of τ and τ ′ implies that the distances are divided by their maximum value, and normalized to [0 , 1] , which increases the robustness of the relation score to the scale of the distances.
ρ is a tunable parameter and set to 0.1 in our experiments, since 0.1 is a commonly used temperature in the softmax function [47, 48].
We denote R sk =
( R sk,i 1 , R sk,i 2 , · · · , R sk,i U ) and R tl =
( R tl,j 1 , R tl,j 2 , · · · , R tl,j U ) that represent the relation of data points x k and y l to the keypoints in source and target domains respectively.
From the definition of the relation, the cross domain points near to a paired keypoints share similar relation to the keypoints in corresponding domain.
For instance, in Fig. 3, x k and y l near to the paired keypoints with indexes ( i 2 , j 2 ) have similar relation R sk and R tl .
Meanwhile, if x k is distant from all the keypoints, R sk,i u is close to a uniform probability vector.
Keypoint-guided model.
Based on the relation given above, we define the guiding matrix G =
( G k,l ) ∈ R m × n by G k,l = d
( R s k , R t l ) where d measures the dissimilarity of R s k and R t l .
Then, G k,l is with small and even near to
if R sk and R tl are similar.
d is taken as the Jensen–Shannon divergence in this paper.
We will study the effect of d in Appendix B.5.
By using the mask-based constraint of transport plan, the KeyPoint-Guided model by ReLation preservation ( KPG-RL ) is defined as
[Equation]
By the KPG-RL model in Eq.
(9), first, the matching of keypoint pairs is enforced by the mask-based constraint of the transport plan.
Second, the minimization of the objective function enforces that the optimal transport plan has larger entries in the locations where the entries of G are smaller.
Hence the cross-domain points corresponding to these locations ( e .
g ., k and l shown in Fig.
3) that are near to a paired keypoints tend to be matched.
Based on the softmax-based formulations in Eqs.
(7) and (8), d ( R sk , R tl ) is mainly determined by the relation score to the closest keypoint(s), since relation scores to the distant keypoints are small or close to 0.
This implies that the points are mainly guided by the closest keypoints in our KPG-RL model in Eq.
For the points distant from all the keypoints, their corresponding entries of G are close to zeros according to the definition of G .
Therefore, the guidance of keypoints to these points is limited.
To achieve correct matching of these points, additional information, e .
g ., point-wise cost or more keypoints, is needed.
Equation (9) is a linear program and can be solved by linear programming algorithms, e .
g .,
the
Simplex algorithm.
We give the details for reformulating Eq.
(9) as the standard form of linear programming in Appendix A.2.
Since Sinkhorn’s algorithm offers a lightspeed computation of the entropy-regularized OT [14], a natural question is that, can Sinkhorn’s algorithm be applied to the
KPG-RL model with entropy regularization?
We give the positive answer, and the details of the deduction are given in Appendix A.3.
The iterative formulas are
[Equation]
[Equation]
where K = M ⊙ e − G/ϵ , ϵ is the coefficient of entropy regularization.
The division operator used above is entry-wise.
After iteration, the optimal transport plan is diag ( u ) K diag ( v ) .
Note that in this KPG-RL model in Eq.
(9), the points in X and Y do not necessarily lie in the same space because we only need to compute the distance within each one of X and Y .
Therefore the proposed KPG-RL model in Eq.
(9) is applicable even when p and q are supported in different spaces.
As mentioned in Sect.
1, the GW model is applicable for transport across different spaces.
However, GW is a non-convex quadratic program and is often solved by the Frank-Walfe algorithm in which a KP-like problem needs to be solved at each iteration by Sinkhorn’s algorithm or linear programming.
Surprisingly, our KPG-RL model can be directly solved using Sinkhorn’s algorithm or linear programming without additional iteration, as discussed above.
Imposing keypoint guidance in KP/GW.
We can impose the keypoint guidance in KP by adding
L kpg ( π ) as a regularization term to KP, obtaining the following KPG-RL-KP model:
[Equation]
The KPG-RL-KP can be solved by Sinkhorn’s algorithm or linear programming, the same as the solution of KPG-RL model.
Similarly, we define the KPG-RL-GW model in GW by
[Equation]
The KPG-RL-GW model is solved using the Frank-Walfe algorithm of which the details are given in Appendix A.4.
The KPG-RL-KP/KPG-RL-GW models take both advantages of KP/GW and
KPG-RL models, and could be helpful especially when the number of paired keypoints is small.
We show in Appendix A.5 that the KPG-RL-KP and KPG-RL-GW provde a proper metric and a divergence under mild conditions.
α is simply set to 0.5 in our experiments.
Extension to partial OT.
We now extend the above KPG-RL model to the more practical partial OT setting that only partial mass is transported.
To do this, we first apply the above mask-based constraint of transport plan to the partial OT model in Eq.
We then add more constraints to enforce that all the mass of keypoints is transported to preserve the matching of keypoints.
Finally, we define the partial-KPG-RL model as
[Equation]
where Π s ( p , q ; M ) = { π ∈ R m
[Equation]
n + |
[Equation]
M
[Equation] [Equation]
M
[Equation]
m ⩽ q ,
[Equation]
m
[Equation]
M ⊙
[Equation] [Equation]
=
s ; ( M ⊙ π ) i, : 1 n = p i , ∀ i ∈ I ; 1 ⊤ m ( M ⊙ π ) : ,j = q j , ∀ j ∈ J } .
Chapel et al .
[20] propose a compelling method to solve the original partial OT problem in Eq.
(2) by transforming the partial OT problem into an OT-like problem.
Inspired by Chapel et al .
[20], to solve our partial-KPG-RL model
[Figure]
in Eq.
(13), we add a dummy point to each of the marginal distributions to transform Eq.
(13) into a
KPG-RL-like problem.
To do this, we extend p , q , G, M by
[Equation]
a
[Equation]
where A > 0 , ξ > 0 , a ∈ R m , b ∈ R n
The element a i of a is 0 if i ∈ I , otherwise 1.
The element b j of b is 0 if j ∈ J , and 1 otherwise.
For more motivations of this extension, please refer to
Appendix A.6.
By the following theorem, solving the optimal transport plan of problem (13) boils down to solving the problem min ¯ π ∈ Π(¯ p , ¯ q ; ¯ M ) ⟨ ¯ M ⊙ ¯ π, ¯ G ⟩ F .
Theorem 1.
Suppose A > 0 , ξ > 0 , (cid:80) i ∈ I p
i < s , and (cid:80)
j ∈ J q j < s , then the optimal transport plan M ⊙ π ∗ of problem (13) is the m -by- n block in the upper left corner of the optimal transport plan ¯ M ⊙ ¯ π ∗ of problem min ¯ π ∈ Π(¯ p , ¯ q ; ¯ M ) ⟨ ¯ M ⊙ ¯ π, ¯ G ⟩ F .
Please refer to Appendix A.7 for the proof of Theorem 1.
Experiments
We evaluate the keypoint-guided models on toy data, and apply them to HDA and open-set HDA.
5.1 Toy Data
As illustrated in Fig.
4, in this toy data experiment, each of the source (blue) and target (green) distributions is a Gaussian mixture composed of three distinct Gaussian components indicated by different shapes where the same shapes indicate points of the same class.
In Fig.
4, we have the following observations.
In Fig. 4(a), in the KP model, the points in each component of target distribution are mismatched to points in different classes of source distributions, and only a small fraction of target points are correctly matched to source points belonging to the same class.
In
Fig. 4(b), given 2 keypoint pairs (from distinct classes), the KPG-RL-KP model apparently improves the correctness of matching.
In Fig. 4(c), the KPG-RL-KP model mainly matches the source points to the target points belonging to the same class, thanks to the given 3 keypoint pairs.
This implies that our proposed keypoint-guided model does help to produce correct matching in OT by leveraging the
[Table]
HOT (w/ mask) 45.
TLB [27] 29.
TLB (w/ mask) 42.
KPG (w/ dist) 55.
KPG-RL-GW 58.
KPG-RL (LP) 56.
KPG-RL (SH) 60.
given a few keypoint pairs.
In Figs. 4(d), 4(e) and 4(f), the proposed KPG-RL-GW model improves the correctness of matching of GW model by leveraging the guidance of given keypoints pairs.
Please refer to Appendix B.1 for the toy experiment for evaluating the partial-KPG-RL model.
5.2 Heterogeneous Domain Adaptation
In HDA, we are given a large amount of labeled source domain data { ( x i , t i ) } mi =1 , a few labeled target domain data { y j , ¯ t j } n l j =1 , and large number of unlabeled target domain data { y j } nj = n l +1 , where m ≫ n l , n ≫ n l , x i and y j are features, and t i and ¯ t j are respectively the class labels of x i and y j .
The heterogeneity means that x i and y j are from different spaces/modalities.
The goal of HDA is to train a classification model using the given data to predict the label of unlabeled target domain data, leveraging the knowledge of large-scale labeled source domain data.
The main challenge is that the domain gap between heterogeneous source and target distributions supported in distinct spaces hinders the direct employment of the source domain trained model in the target domain.
We tackle the problem of HDA using our proposed KPG-RL model as follows.
We first transport the source domain data using our KPG-RL model to the target domain.
More concretely, in KPG-RL, each labeled target domain data and the source domain class center of the same class are taken as a matched keypoint pair.
The KPG-RL model is then performed between empirical distributions of the source domain data along with class centers and the target domain data (including labeled and unlabeled data).
Based on the produced optimal transport plan, we transport the source domain data using the barycentric mapping [49] to the target domain.
Finally, we train the classification model (taken as a kernel SVM) on the transported source domain data (using their class label before transport) and the labeled target domain data, which is applied to the target domain test data.
ϵ is set to 0.005.
More details of the barycentric mapping, the kernel SVM, etc , are given in Appendix B.4.
We compare our method with the following baseline methods, including 1) “Labeled-target-only” that trains the kernel SVM using the labeled target data; 2) the OT methods of “GW”, “HOT”, and “TLB” that transport source domain data using barycentric mapping induced by the transport plan of
GW [27], Hierarchical OT
[33], and TLB [27], and then train the kernel SVM on the transported data and labeled target domain data; 3) the typical HDA methods of “SGW” [37] and “STN” [50], and the recent HDA methods of “SSAN” [51], “DDACL” [52] and“CDSPP” [45].
We conduct experiments on Office-31 [53] dataset.
On Office-31, we use the DeCAF 6 [54] features and the features extracted by ResNet-50 [55] pretrained on ImageNet [56] to respectively build source and target domains for constructing heterogeneous adaptation tasks.
In each task, one labeled data (1-shot) for each class is
[Table]
Table 1 reports the results on Offce-31 dataset.
In Table 1, “KPG-RL (SH)” and “KPG-RL (LP)” denote our KPG-RL model solved using Sinkhorn’s algorithm and linear programming respectively.
KPG-RL (SH) achieves marginal better average accuracy than KPG-RL (LP), which could be because the barycentric mapping based on the more dense transport plan optimized by Sinkhorn’s algorithm is more robust to wrong matching.
We also observe that KPG-RL-GW achieves slightly lower average accuracy than KPG-RL (SH), and KPG-RL (SH) is more computationally efficient as discussed in Sect. 4.
“KPG (w/ dist)” is the approach that imposes the guidance of keypoints using distance preservation in our framework, i .
e .
, R sk,i u and R tl,j u in Eqs.
(7) and (8) are taken as C sk,i u and
C tl,j u respectively.
KPG-RL (SH) improves the accuracy of KPG (w/ dist) by 15%, implying that our relation-preserving scheme is more effective for imposing the guidance of keypoints than the distance-preserving scheme.
Table 1 shows that the results of GW, HOT, and TLB are inferior to that of Labeled-target-only, indicating that GW, HOT, and TLB cause negative transfer.
This is reasonable because GW, HOT, and TLB do not take advantage of the guidance of keypoints and can cause wrong matching.
We also use our mask-based constraint on the transport plan in GW, HOT, and
TLB, of which the corresponding approach are denoted as GW (w/ mask), HOT (w/ mask), TLB (w/ mask).
We can observe that with the mask-based constraint, the performances of GW, HOT, and
TLB are improved but worse than KPG-RL (SH) and KPG-RL (LP), verifying the importance of the relation preservation scheme in KPG-RL.
SGW [37] aligns the class centers of labeled target domain data and transported source domain data in the GW model, and realizes positive transfer.
Our proposed KPG-RL (SH) outperforms SGW by 12.2%, confirming that our KPG-RL model preserving relation is more effective than SGW preserving class centers for HDA.
Compared with the other HDA methods, KPG-RL (SH) achieves the best average accuracy.
The keypoints are important to KPG-RL, since they are used to guide the matching.
We show the results with varying keypoints by giving different numbers and sampling of labeled target domain data, in Table 2.
It can be observed in Table 2 that under different numbers and samplings of labeled target domain data, our approach KPG-RL (SH) consistently outperforms the baseline Labeled-target-only, achieving positive transfer.
We also find that as the number of labeled target domain data increases, the margin between the accuracy of Labeled-target-only and KPG-RL (SH) decreases.
This indicates that given more labeled target domain data, the necessity of knowledge transferred from source domain becomes smaller.
Please refer to Appendix B.5 for the effect of source keypoints.
5.3 Open-Set Heterogeneous Domain Adaptation
We conduct open-set HDA experiments to evaluate our proposed partial-KPG-RL model.
The problem setting of open-set HDA is the same as HDA, except that the unlabeled target domain data contain samples of unknown classes absent in the categories of labeled data.
The goal of open-set
HDA is to correctly classify the common class samples and to detect the unknown class samples.
We consider the case that the proportion η of unknown class samples in the unlabeled target domain data is given.
For applying partial-KPG-RL to open-set HDA, the keypoint pairs are constructed using labeled target domain data and their corresponding source class centers, same as KPG-RL for HDA in Sect.
We use the partial-KPG-RL model in Eq.
(13) to transport all the labeled source domain data to partially match the target domain data.
The unmatched target domain data are detected as unknown class.
We then train a kernel SVM on the transported source domain data and labeled target domain data to classify the common class samples detected by partial-KPG-RL.
The natural Baseline for open-set HDA is the approach that rejects the η -proportion unlabeled samples with largest distance to labeled target domain data as unknown, and trains a kernel SVM on the labeled target domain data to classify common class data.
We also evaluate the HDA methods STN [50] and SSAN [51] in the
[Table]
open-set HDA task.
For STN [50] and SSAN [51], we reject the η -proportion unlabeled samples with lowest prediction confidence as unknown class.
Table 3 reports the results.
We use the open-set evaluation metrics [57] including accuracy of common class sample (OS ∗ ), accuracy of unknown class sample (UNK), and their harmonic mean HOS = 2 OS ∗ × UNK OS ∗ + UNK.
We can observe in Table 3 that our proposed partial-KPG-RL achieves the best results in terms of average OS ∗ , UNK, and HOS, indicating that the partial-KPG-RL is effective for both classifying common class data and identifying unknown class data.
Compared with the Baseline, partial-KPG-RL outperforms it by 20.4% in terms of average HOS, confirming the positive transfer achieved by our method.
More implementation details and the solution for Open-Set HDA with unknown η are given in Appendix B.2.
Due to space limit, we include the experiment of our approach for deep unsupervised domain adaptation in Appendix B.3.
More empirical analysis and ablation studies, e .
g ., sensitivity to hyperparameters, the effect of d , the discussion on how to define the keypoints in more general practical applications, the time and memory cost, etc. , are given in Appendix B.5.
Conclusion
In this paper, we propose a novel KPG-RL model that leverages keypoints to guide the correct matching in OT.
We devise a mask-based constraint to preserve the matching of keypoints pairs, and propose to preserve the relation of each point to the keypoints to impose the guidance of these keypoints in OT.
The effectiveness of the proposed KPG-RL model is verified in the application of HDA.
The keypoint-guided OT model could be possibly applied to more applications, e .
g ., cross-domain data generation/translation [58, 59], and point-set or image registration.
Acknowledgements.
This work was supported by National Key R&D Program 2021YFA and NSFC (12125104, U1811461, U20B2075, 11971373, 12026605).
Checklist
For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
[Yes] See Abstract and Introduction.
(b) Did you describe the limitations of your work?
[Yes] See the end of the first paragraph on page 6.
(c) Did you discuss any potential negative societal impacts of your work?
[N/A]
This work does not present any significant societal consequence.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] We have read them.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[Yes] See Propositions 1 and A-1, and Theorems 1, A-1, A-2, A-3.
(b) Did you include complete proofs of all theoretical results?
[Yes] See Appendix A. 3.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] See Sect.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] See Sect.
5 and Appendix B. (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[No] We have reported the results for different samples in Table 2.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[N/A]
The experiments are conducted on CPU.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[Yes] See Sect.
(b) Did you mention the license of the assets?
[N/A]
The datasets are public.
(c) Did you include any new assets either in the supplemental material or as a URL?
[No] (d) Did you discuss whether
and how consent was obtained from people whose data you’re using/curating?
[N/A] (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
[N/A] 5.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]
A Mathematical Deductions A.1 Proof of Proposition
Proposition 1 in the paper is for the case that p i =
q j , for all ( i, j ) ∈ K .
As stated in the paper, the mask-based modeling of the transport plan is applicable even for the case that there exist some
3 6 3 6𝒑𝒑 𝒒𝒒 𝑞𝑞 1 𝑞𝑞 2 𝑞𝑞 3 𝑞𝑞 4 𝑞𝑞 5 𝑞𝑞𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝    �𝜋𝜋 0 0�𝜋𝜋0  0 0
�𝜋𝜋 �𝜋𝜋 𝑀𝑀 𝜋𝜋 𝒑𝒑 𝒒𝒒 1 0 1 0 11 0 1 0 110 10 0 0 1 01 0 1 0 10 0 0 0
( i, j ) ∈ K such that p i ̸
= q j .
To see this, we first mathematically give the definition of preserving the matching of keypoint pairs and then prove Proposition A-1, a generalization of Proposition 1.
Definition A-1.
Given the marginal distributions p and q , we say that the transport plan π ∈ Π( p , q ) preserves the matching of a keypoint pair with index ( i, j ) ∈ K , if π satisfies one of the following conditions:
If p i =
q j , π satisfies that π i,j ′ = 0 , ∀ j ′ ̸
= j ; π i ′ ,j = 0 , ∀ i ′ ̸
= i ; π i,j = p i =
q j . 2.
If p i > q j , π satisfies that π i ′ ,j = 0 , ∀ i ′ ̸
= i ; π i,j = q j . 3.
If p i < q j , π satisfies that π i,j ′ = 0 , ∀ j ′ ̸
= j ; π i,j = p i .
The left part of Fig.
A-1 illustrates these conditions.
Specifically, the first condition implies that if p i =
q j ( e .
g .
, ( i, j ) is taken as (4, 4) in Fig.
A-1)
, the all mass p i of x i will be transported to y j and y j can only receive mass from x i .
The second condition implies that if p i > q j ( e . g .
, ( i, j ) is taken as (3, 2) in Fig. A-1), y j can only receive mass from x i and consequently the partial mass p i − q j of x i is allowed to be transported to the target points apart from y j .
The third condition indicates that if p i < q j ( e .
g .
, ( i, j ) is taken as (6, 5) in Fig.
A-1)
, the all mass p i of x i will be transported to y j and y j is enabled to receive partial mass q j − p i from the source points apart from x i .
For the convenience of description, for each pair ( i, j ) ∈ K , we denote j = κ ( i ) and i = κ ′ ( j ) .
Proposition A-1.
Suppose that the mask matrix M satisfies that
[Equation]
Then, the transport plan ˜ π
= M ⊙ π with π ∈ Π( p , q ; M ) preserves the matching of keypoint pairs with index in K .
According to the definition of M , M i,j = 1 for the keypoint pair ( i, j ) ∈ K , implying that ˜ π
i,j could take non-zero value.
For i ∈ I , ( i, j ) / ∈ K and p i ≤ q κ ( i ) , M i,j is set to 0, enforcing that the i -th row of ˜ π are zeros except for the location κ ( i ) of the target keypoint paired with i ( e .
g ., the 4-th and 6-th rows of ˜ π in Fig. A-1).
Similarly, for j ∈ J , ( i, j ) / ∈ K , and p κ ′ ( j ) ≤ q j , we set M i,j = 0 , enforcing that the j -th column of ˜ π are zeros except for the location κ ′ ( j ) of the source keypoint paired with j ( e .
g ., the 2-th and 4-th columns of ˜ π in Fig.
A-1).
For the other points (corresponding to the last three cases in Eq.
(A-1))
, we set M i,j = 1 , indicating that there is no additional constraint on ˜ π i,j .
If p i =
q j , for all ( i, j ) ∈ K ( i .
e .,
p i =
q κ ( i ) , ∀ i ∈ I ), Proposition A-1 degenerates to Proposition 1 in the paper.
Proof
For any ( i, j ) ∈ K , we next prove that ˜ π preserves the matching of keypoint pair ( i, j ) .
• If p i = q j , from the definition of M , we have M i,j ′ = 0 for all j ′ ̸
= j and M i,j = 1 .
Then, we have ˜ π i,j ′ =
M i,j ′ π
i,j ′ = 0 for all j ′ ̸ = j .
Since (cid:80) nj ′ =
1 ˜ π i,j ′ =
p i , we have ˜
π
i,j = p i .
Similarly, we have M i ′ ,j = 0 for all i ′
̸
= i .
Then, we have ˜ π
i ′ ,j = M i ′ ,j π i ′ ,j = 0 for all i ′ ̸
= i , and ˜ π i,j = (cid:80) mi ′ =
1 ˜ π i ′ ,j = q j . •
If p i > q j , from the definition of M , we have M i ′ ,j = 0 for all i ′
̸
= i and M i,j = 1 .
Then, we have ˜ π
i ′ ,j = M i ′ ,j π i ′ ,j = 0 for all i ′ ̸
= i , and ˜ π i,j = (cid:80) mi ′ =
1 ˜ π i ′ ,j = q j . • p
i < q j , from the definition of M , we have M i,j ′ = 0 for all j ′ ̸
= j and M i,j = 1 .
Then ˜
π i,j ′ =
M i,j ′ π
i,j ′ = 0 for all j ′ ̸
= j , and ˜ π i,j = (cid:80) nj ′ =1 ˜ π i,j ′ =
q i .
Thus, for any keypoint pair with index ( i, j ) ∈ K , ˜ π satisfies the conditions in Definition A-1.
This means that ˜ π preserves the matching of keypoint pairs with index in K .
A.2 Linear Programming for Solving KPG-RL
We cast the matrix G (resp.
M, π ) as the vector c (resp.
m , x ) ∈ R mn , such that the ( i + m ( j − 1) )
-th element of c is G ij .
By denoting
[Equation]
where I m is the identity matrix of size n and ⊗ is the Kronecker product, the KPG-RL model in
Eq.
(9) in the paper reads
[Equation]
With the standard form of linear programming in Eq.
(A-3), the Simplex algorithm can be directly used to solve the KPG-RL model.
A.3 Sinkhorn’s Algorithm for Solving KPG-RL
The entropy-regularized model for KPG-RL is
[Equation]
where H ( M ⊙ π ) = − ( ⟨ M ⊙ π, log( M ⊙ π ) ⟩ F − 1 ⊤ m ( M ⊙ π ) 1 n ) is the entropy of the transport plan M ⊙ π .
The Lagrangian function is
[Equation]
where f ∈ R m and g ∈ R n .
The first-order conditions then yield
[Equation]
If M i,j = 0 , π i,j could be arbitrary non-negative value, and if M i,j = 1 , we have π i,j = e f i /ϵ
e − G i,j /ϵ e g j /ϵ .
Therefore, we can unify the expression as π ij =
M ij e f i /ϵ
e − C ij /ϵ
e g j /ϵ , in which we enforce π i,j =
if M i,j = 0 .
The matrix form is π = diag ( u ) K diag ( v )
where u = e f /ϵ , K
= M ⊙ e − G/ϵ , and v = e g /ϵ .
The constraints are
[Equation]
Since the entries of K are non-negative, the Sinkhorn’s algorithm can be applied [15].
The iteration formulas are
[Equation]
The division operator used above is entry-wise.
Log-domain Sinkhorn iteration.
For KP, the Sinkhorn iteration in the log-domain is more stable [60].
We next deduce the log-domain Sinkhorn iteration for our KPG-RL.
In the log-domain, the left equation in Eq.
(A-8) is
[Equation]
Let H ( f , g ) =
1 ϵ ( − G + f 1 ⊤ n + g 1 ⊤ m ) , then
[Equation]
If M i,j = 0 , log( M i,j ) H ( f ( l ) , g ( l ) )
i,j = −∞ .
We define ¯ H ( f , g ) as
[Equation]
if
[Equation]
(A-11) and define the log-sum-exp function LogSumExp : R m × n → R m as
LogSumExp
[Equation]
A
[Equation] [Equation]
Then, the matrix form of Eq.
(A-10) becomes
[Equation]
Similarly, the corresponding iteration formula in the log-domain of the right equation in (A-8) is
[Equation]
Equations (A-13) and (A-14) consists in the formulas of the Sinkhorn iteration in the log-domain.
A.4 Frank-Walfe Algorithm for Solving KPG-RL-GW
We define the 4-order tensor L =
( L i,j,k,l ) ∈
R m × n × m × n by L i,j,k,l = ( C si,k − C tj,l ) 2 , and define the tensor-matrix product L ◦
π =
(( L ◦ π ) i,j
) ∈
R m × n by ( L ◦ π ) i,j = (cid:80) k,l L i,j,k,l π k,l .
Then,
The
KPG-RL-GW model in Eq.
(12) in the paper reads
[Equation]
The gradient of the objective function is
[Equation]
1 − α ) G ) .
(A-
[Equation]
In k -th iteration of the Frank-Walfe algorithm, it runs the following three steps:
Step 1.
Compute a linear minimization oracle over the set Π( p , q ; M ) ,
i
e .,
[Equation]
Equation (A-17) can be rewritten as
[Equation]
Equation (A-18) is a KPG-RL-like problem and can be solved using linear programming or Sinkhorn’s algorithm.
Step 2.
Determine optimal step-size β ( k ) subject to
[Equation]
β ( k ) can be obtained by the line-search method in [61].
Step 3.
Update
[Equation]
A.5 Theoretical Properties of KPG-RL-KP and KPG-RL-GW
In this section, we show that given prior “correct” paired keypoints, the KPG-RL-KP model provides a proper metric for distributions supported in the same space, and the the KPG-RL-GW model provides a divergence for distributions in distinct spaces, under mild conditions.
Since the discrete distributions p = 1 m (cid:80) mi δ x i (resp.
q = 1 n (cid:80) nj δ y j ) are invariant to the permutation of { x i } mi = (resp.
{ y j } nj =1 ), we assume that any two paired keypoints across domains share the same index.
Therefore, the index set of paired keypoints is K
=
( i u , i u ) }
Uu =1 .
We assume p i u
=
q i u , ∀ i u , in this section.
For the convenience of description, in this section, we denote M pq as the mask matrix for transporting p to q , and P XI as the set of discrete probability distributions supported on m points in ground space X such that all distributions in P XI share the keypoint index set I
= { i u } Uu =1 .
A.5.
KPG
RL
KP Providing a Proper Metric
For distributions supported in the same space, the “correct” paired keypoints indicates that if p = q , each source keypoint is equal to its paired target keypoint, i .
e ., x i u
= y
i u , for any i u ∈
I .
We denote
[Equation]
where α ∈ (0 , 1) .
Theorem A-1.
Suppose c is a proper distance in space X and d is a proper distance in probability simplex Σ m .
Then, for any p and q in P XI , given the “correct” paired keypoints stated above,
S krk ( p , q ) is a proper distance between p and q .
Proof : (1) Show that S krk ( p , q ) =
if and only if p = q .
(a) If p = q , we have x i =
y i and p i
= q i for any i ∈ [ m ] (since the permutation of support points does not change the distribution).
Hence,
C i,i =
c ( x i , y i )
= 0 , and C si,
i u
= c ( x i , x i u )
=
c ( y i , y i u ) =
C ti,i u , ∀ i ∈ [ m ] and ∀ i u ∈ I , which implies that R si = R ti .
Then, we have G i,i =
d ( R si , R ti ) = 0 .
We define π by π i,j = p i if i = j , and 0 otherwise.
Obviously, M pq ⊙ π is in Π( p , q ; M pq ) and (cid:80) i,j M pq i,j π i,j ( αC i,j + (1 − α )
G i,j ) = 0 .
Therefore, S krk ( p , q ) = 0 .
(b) We denote π ∗ as the optimal solution of problem (A-21).
If S krk ( p , q ) = 0 , we have ⟨ M pq ⊙ π ∗ , C ⟩ F = 0 .
This means that the KP problem min π ∈ Π( p , q ) ⟨ π, C ⟩ F = 0 .
Using the Proposition 2.2 in [62], we have p = q .
(2) Show that S krk ( p , q )
=
S krk ( q , p ) .
From the definition of mask matrix in Proposition 1 in the paper, we have M pq i,j = M
qp j,i .
C and G are symmetric because c and d are distances.
For any π ∈ Π( p , q ; M pq ) , we define π ′ as π ′ i,j = π j,i , and then π ′ ∈ Π( q , p ; M qp ) .
Then, we have
[Equation]
(3) Show that S krk ( p , q ) ≤ S krk ( p , r )
+ S krk ( r , q ) .
Let M pr ⊙ π pr and M rq ⊙ π rq be the optimal transport plans corresponding to S krk ( p , r ) and S krk ( r , q ) , respectively.
We define
[Equation]
where the element ˜ r j of ˜ r is r j if r j > 0 , and 1 otherwise.
We notice that
[Equation]
where the j -th location of ˜ 1 m is 1 if r j > 0 , and 0 otherwise.
Note that for j such that r j = 0 , we have (cid:80) i,j ( M pr ⊙ π pr ) i,j = r j = 0 , which implies ( M pr ⊙ π pr ) i,j = 0 for any i .
Hence,
[Equation]
Similarity, ˜ γ ⊤ 1 m = q .
Since the indexes of paired keypoints across any two distribution in P XI are the same, for any
i u ∈ I , the i u -th row and column of M pr and M rq are zeros except for that
M pr i u ,i u
=
M rq i u ,i u
= 1 .
So the i u -th row and column of ˜ γ are zeros except for ˜ γ i u ,i u .
Then, we can write ˜ γ i u ,i u
= M pq ⊙ γ with γ ∈ R m × m + .
Further, we have γ ∈ Π( p , q ; M pq ) .
The triangle inequality follows then from
[Equation]
i
[Equation]
where z k is the support point of r and R rk is the relation of z k to the keypoints of r .
A.5.2 KPG-RL-GW Providing a Divergence
For any distribution p ∈ P XI and q ∈ P YI , p and q are said to be isomorphic if there exists a bijection σ : [ m ] (cid:55)−→ [ m ] such that c ( x i , x k ) =
c ′ ( y σ ( i ) , y σ ( k ) ) , and p i =
q σ ( i ) , where [ m ] = { 1 , 2 , · · · , m } , and c and c ′ are respectively proper distances in X and Y .
The keypoints are “correct” means that if p = q , σ maps each source keypoint to its paired target keypoint, i .
e ., σ ( i u ) =
i u , for any i u ∈
I .
We denote
[Equation]
Theorem A-2.
Suppose c and c ′ are proper distances in spaces X and Y .
Suppose d is a divergence in probability simplex Σ m .
Then, for any p in P XI and any q in P YI , given the “correct” paired keypoints stated above, S krg ( p , q ) =
if and only if p and q are isomorphic.
Proof:
(a) If p and q are isomorphic, for any i ∈ [ m ] and
any i u ∈ I , we have c ( x i , x i u ) =
c ′ ( y σ ( i ) , y σ ( i u ) )
=
c ′ ( y σ ( i ) , y i u ) , implying that R si = R tσ ( i ) .
We define π as π i,j = p i
if j = σ ( i ) , and 0 otherwise.
We then have
(cid:88)
[Equation]
This implies S krg ( p , q ) = 0 .
(b) Let ( M pq ) ⊙ π ∗ be the optimal transport plan corresponding to S krg ( p , q ) = 0 .
If S krg ( p , q ) = 0 , we have
[Equation]
This indicates that the Gromov-Wasserstein distance
[Equation]
By virtue to Gromov-Wasserstein properties in [27], there exists a bijection σ : [ m ] (cid:55)−→ [ m ] such that c ( x i , x k ) =
c ′ ( y σ ( i ) , y σ ( k ) ) , and p i =
q σ ( i ) .
From Theorem A-2, the KPG-RL-GW model provides a divergence in the sense of isomorphism.
A.6 Motivations of the Solving Algorithm for Partial-KPG-RL Model
In the partial-KPG-RL model in Eq.
(13) in the paper, only s -unit mass of source and target distributions is matched.
Inspired by [20], we add a dummy point with mass ∥ q ∥ 1 − s for source domain (the left black circle in Fig. A-2) and a dummy point with mass ∥ p ∥ 1 − s for target domain (the right black circle in Fig. A-2).
We denote ¯ p
= ( p ⊤ , ∥ q ∥ 1 − s ) ⊤ and ¯ q = ( q ⊤ ,
∥ p ∥ 1 − s ) ⊤ .
As illustrated in Fig.
A-2, we aim to design the extended guiding matrix ¯ G and extended mask matrix ¯ M such that performing KPG-RL between ¯ p and ¯ q will transport ∥ p ∥ 1 − s mass from source real data points to the target dummy point and transport ∥ q ∥ 1 − s mass from source dummy point to target real data points.
As a sequence, only s mass of source and target real data points are matched.
Meanwhile, the keypoints should not be matched to the dummy points because they are annotated data to guide the matching.
To do this, we extend G, M by
[Equation]
where A > 0 , ξ > 0 , a ∈ R m , b ∈ R n , and M is constructed as in Proposition A-1.
The element a i of a is 0 if i ∈ I , and 1 otherwise.
The element b j of b is 0 if j ∈ J , and 1 otherwise.
By
Theorem A-3, solving partial-KPG-RL model boils down to solving the KPG-RL-like problem
[Equation]
[Figure]
[Equation]
[Figure]
[Equation]
[Figure]
A.7 Proof of Theorem
For the convenience of understanding, the Theorem 1 in the paper is for the case that p i =
q j for all ( i, j ) ∈ K .
In this appendix, we provide and prove the Theorem A-3, a generalization of Theoremin the paper, for the general case that there could exist some ( i,
j ) ∈ K such that p i ̸ = q j .
Before that, we rewrite the partial-KPG-RL model first.
Partial-KPG-RL model:
[Equation]
where Π s ( p , q ; M ) = { π ∈ R m × n + |
[Equation]
s ; ( M ⊙ π ) i, : 1 n = p
i , ∀ i ∈ I ; 1 ⊤ m ( M ⊙ π ) : ,j = q j , ∀ j ∈ J } .
Theorem A-3.
Suppose A > 0 , ξ > 0 , (cid:80) i ∈I p
i + max { q κ ( i ) − p i , 0 } < s , and (cid:80) j ∈J q j + max
p κ ′ ( j ) − q j , 0 } < s , then the optimal transport plan M ⊙ π ∗ of partial-KPG-RL model is the m -by- n block in the upper left corner of the optimal transport plan ¯ M ⊙ ¯ π ∗ of problem min ¯ π ∈ Π(¯ p , ¯ q ; ¯ M ) ⟨ ¯ M ⊙ ¯ π, ¯ G ⟩ F .
The definitions of κ ( i ) and κ ′ ( j ) are given in Appendix A.1.
The condition (cid:80) i ∈I p
i + max { q κ ( i ) − p i , 0 } <
s implies that the sum of the mass ( (cid:80) i ∈I p i ) of source keypoints and the mass ( (cid:80) i ∈I max { q κ ( i ) − p
i , 0 } ) of the other source points apart from keypoints that should be transported to target keypoints is less than s .
The condition (cid:80) j ∈J q j + max { p κ ′ ( j ) − q j , 0 } < s implies that the sum of the mass ( (cid:80) j ∈J q j ) of target keypoints and the mass ( (cid:80) j ∈J max { p κ ′ ( j ) − q j , 0 } ) of the other target points apart from keypoints received from source keypoints is less than s .
The two conditions are reasonable to guarantee the admissible solutions of problem (A-31).
If p i =
q j for all ( i, j ) ∈ K ( i .
e .,
p i =
q κ ( i ) , ∀ i ∈ I and q j = p κ ′ ( j ) , ∀ j ∈ J ), Theorem A-3 degenerates to the
Theorem 1 in the paper.
A.7.1 Proof
We denote ¨ π =
¯
π ∗ 1: m, 1: n and t =
¯ π ∗ m +1 ,n +1 .
To prove Theorem A-3, we first give some preparations, and then conduct the following three steps.
In step 1, we show that t = 0 .
In step 2, we show that
¨ π ∈ Π s ( p , q ; M ) which means that ¨ π is a feasible solution of problem (A-31).
In step 3, we show that M ⊙ ¨
π is the optimal transport plan of problem (A-31).
We next detail these steps.
Preparations:
Since ¯ π ∗ ∈ Π(¯ p , ¯ q ; ¯ M ) and t = ¯ π ∗ m +1 ,n +1 , we have
[Equation]
Meanwhile, we have
[Equation]
and
[Equation]
A
[Equation]
Combining the above four equations, we have
[Equation]
A
[Equation]
Therefore,
[Equation]
A
[Equation] [Equation]
Step 1: show that t =
¯ π ∗ m +1 ,n +1 = 0 .
First, we have
[Equation]
Suppose ¯ π ∗ m +1 ,n +1 > 0 , we next construct a solution γ such that γ m +1 ,n +1 = 0 and leads to conflict.
We randomly select a set S
= { ( i, j ) | ¯ π ∗ i,j > 0 , i ≤ m, j ≤ n, i / ∈ I , j / ∈ J } and a index pair ( i 0 , j 0 ) satisfying the constraints of elements in S , such that (cid:80) ( i,j ) ∈
S ¯ π ∗
i,j ≤ t and (cid:80) ( i,j ) ∈
S ¯ π ∗
i,j + ¯ π ∗ i 0 ,j 0 > t .
In the rest part of this section, the involved i, j satisfy i ≤ m and j ≤ n .
Such non-empty S and
( i 0 , j 0 ) always exist, because
[Equation]
1 ⊤
[Equation]
M
[Equation]
¨
[Equation]
) 1 n = s + t , and (cid:80) i ∈I p
i
[Equation]
max { q κ ( i ) − p
i
[Equation]
0 } < s , we have (cid:80) i/ ∈I
[Equation]
j/ ∈J ¯ π ∗ ij > t .
We now move the mass of index pairs in S and ( i 0 , j 0 ) to their marginal such that a total mass of t is moved.
Specifically, for ( i, j ) ∈ S , we set γ i,j = 0 , γ i,n +1 =
¯ π ∗ i,n +1 + ¯ π ∗ i,j , γ m +1 ,j = ¯ π ∗ m +1 ,j + ¯ π ∗ i,j .
For ( i 0 , j 0 ) , we set γ i 0 ,j 0 =
¯ π ∗ i 0 ,j 0 − ( t − (cid:80) i/ ∈I ,
j/ ∈J
¯ π ∗ i,j ) , γ i 0 ,n +1 =
¯ π ∗ i 0 ,n +1 + ( t − (cid:80) i/ ∈I ,j/ ∈J
¯ π ∗ i,j ) , γ m +1 ,j 0 = ¯ π ∗ m +1 ,j 0 − ( t − (cid:80) i/ ∈I ,j/ ∈J
¯ π ∗ i,j ) .
For ( i, j ) / ∈ S , we set γ i,j =
¯ π ∗ i,j , γ i,n +1 =
¯ π ∗ i,n +1 , γ m +1 ,j = ¯ π ∗ m +1 ,j .
It is easy to verify that γ ∈ Π(¯ p , ¯ q ; ¯ M ) .
Similar to
Eq.
(A-38), we have
[Equation]
Using the optimality of ¯ M ⊙ ¯ π ∗ , we have
[Equation]
[Equation]
From the definition of γ , we can see that γ i,j ≤ ¯ π ∗ i,j , and thus (cid:80) mi =1 (cid:80) nj
=1 M i,j ( γ i,j − ¯ π ∗ i,j ) G i,j ≤ 0 .
Hence, from Eq. (A-41), we have A < 0 , contradicting the assumption that A > 0 .
Therefore, t = ¯ π ∗ m +1 ,n +1 = 0 holds.
Step 2: show that ¨ π is a feasible solution of problem in Eq.
(A-31) .
We verify the constraints as follows.
(1) Since ¯ π ∗ ≥ 0 , we have ¨ π ≥ 0 .
(2) ( ¯ M ⊙ ¯ π ∗ ) 1 n +1 =
(cid:20) M ⊙ ¨ π a ⊙ ¯ π ∗ 1: m,n +1 b ⊙ ¯ π ∗ m +1 , 1: n (cid:21) (cid:20) 1 n (cid:21) =
(cid:20) p
∥
q ∥ 1 − s (cid:21) , then ( M ⊙ ¨ π ) 1 n + a ⊙ ¯ π ∗ 1: m,n +1 = p , and ( M ⊙ ¨ π ) 1 n ≤ p .
(3) Similarly, from 1 ⊤ m +
( ¯ M ⊙ ¯ π ∗ ) =
( q , ∥ q ∥ 1 − s ) ⊤ , we have 1 ⊤ m ( M ⊙ ¨ π ) ≤ q .
(4) 1 ⊤ m ( M ⊙ ¨ π ) 1 n = s holds, because t = 0 as in Step 1.
(5) ∀ i ∈
I , ( ¯ M ⊙ ¯ π ∗ )
i, : 1 n +1 =
( M ⊙ ¨ π ) i, : 1 n + a i ¯ π ∗ i,n +
= p i .
Since a i =
we have
( M ⊙
¨ π ) 1 , : 1 n = p i .
(6) ∀ j ∈ J , 1 ⊤ m +
( ¯ M ⊙ ¯ π ∗ ) : ,j = 1 ⊤ m ( M ⊙ ¨ π ) : ,j + b j ¯ π ∗ m +1 ,j = q j .
Since b j = 0 , we have
1 ⊤ m ( M ⊙ ¨ π ) : ,j = q j .
Therefore, we have ¨ π ∈ Π s ( p , q ; M ) , and ¨ π is a feasible solution of problem in Eq.
(A-31).
Step 3: show that M ⊙ ¨ π is the optimal transport plan of problem in Eq.
(A-31) .
Suppose there exist a transport plan M ⊙ γ with γ ∈ Π s ( p , q ; M ) such that
[Equation]
We construct ¯ γ as follows.
For i ≤ m, j ≤ n , ¯ γ
i,j = γ i,j .
¯ γ i,n +1 = p
i − (cid:80) nj =1 γ
i,j , ∀ i ≤ m .
¯ γ m +1 ,j = q j − (cid:80) ni
=1 γ i,j , ∀ j ≤ n .
¯ γ m +1 ,n +1 = 0 .
Easily, we can verify that ¯ γ is in Π(¯ p , ¯ q ; ¯ M ) .
Meanwhile,
[Equation]
(A-42)
This contradicts the fact that ¯ M ⊙ ¯ π ∗ is the optimal transport plan of problem min ¯ π ∈ Π(¯ p , ¯ q
[Equation]
¯ M ) ⟨ ¯ M ⊙ ¯ π, ¯ G ⟩ F .
Therefore, M ⊙ ¨ π is the optimal transport plan of problem in Eq.
(A-31).
B Additional Experimental Details and Results B.1 Toy Experiment for Evaluating Partial-KPG-RL model
[Figure]
Figure A-3 illustrates the toy data experiment for evaluating the partial-KPG-RL model.
In Fig.
A-3, the source (blue) and target (green) distributions are Gaussian mixtures.
The source (resp.
target) distribution is composed of three (resp.
two) distinct Gaussian components indicated by different shapes where the same shapes indicate the points of the same class.
When conducting OT, the source class data represented by “ △ ” should not be transported.
In Figs.
A-3(a) and A-3(b), we can observe that both the partial-OT model (defined in Eq.
(2) in the paper) and the partial-GW model [20] wrongly transport some source points of class “ △ ” to target domain and lead to low matching accuracy.
With the guidance of a few keypoints (red pairs), our proposed partial-KPG-RL model does not transport the source points of class “ △ ” to target domain and apparently improves the matching accuracy as in Fig.
A-3(c).
B.2 Additional Experimental Details and Results for Open-set HDA
More experimental details.
In open-set HDA, we are given a large amount of labeled source domain data { ( x i , t i ) } mi =1 , a few labeled target domain data { y j , ¯ t j } n l j =1 , and a large number of unlabeled target domain data { y j } nj = n l +1 .
The fraction of unknown class data is η .
To apply the partial-KPG-RL model defined in Eq.
(13) in the paper to open-set HDA, for each labeled target domain data, we take its corresponding source class center to construct a keypoint pair.
We then resample the source domain data such that the total number of resampled source domain data and the source keypoints is m ′ =
(1 − η ) n .
We define the source distribution as p = 1 − ηm ′
( (cid:80) n l j =1 δ
c
j
+ (cid:80) m ′
j = n l +1 δ x ′ j ) , where x ′ j is a resmapled source domain sample and c j is the source class center corresponding to the target labeled sample y j .
The target distribution is defined as q = 1 n (cid:80) nj =1 δ y j .
[Table]
The partial-KPG-RL model is conducted to transport mass from p to q with s = 1 − η .
After transport, the η -fraction unlabeled target data receiving smallest mass from source domain are detected as unknown class and the rest unlabeled target data are taken as common class ones.
Finally, we train the kernel SVM on the transported source domain data and labeled target domain data to classify the unlabeled target domain common class data.
Results for open-set HDA with unknown η .
For the more practical open-set HDA setting that η is unknown, researchers can design methods to estimate η and then apply our method using the estimate of η , or take η as a hyper-parameter and design methods to tune it.
We directly use the positive-unlabeled learning [63] method [64] to estimate the fraction of common class data among the target domain unlabeled data, by taking the labeled target data as positive samples.
The results of different methods for open-set HDA using the estimate ˆ η of η are given in Table A-1.
According to
Table A-1, the positive transfer is achieved by our method.
We can see that ˆ η in all tasks is lower than the true η , implying that less unknown class samples are detected.
Correspondingly, the UNK value (73.2%) achieved by partial-KPG-RL using ˆ η in Table A-1 is smaller than that (83.5%) using η in Table 3 in the paper.
Surprisingly, the OS ∗ value (67.5%) of partial-KPG-RL in Table A-1 is higher than that (59.7%) in Table 3 in the paper.
As a balance, the HOS value (69.5%) achieved by partial-KPG-RL using ˆ η is similar to the HOS value (69.1%) of partial-KPG-RL using the true η .
In the following Table A-2, we take η as a hyper-parameter and show the average HOS achieved by partial-KPG-RL using varying magnitude of η .
It is observed that the average HOS is stable to η in a relatively large range of [0 . 50 , 0 . 80] .
[Table]
.3 Application in Deep Unsupervised Domain Adaptation
In this section, we apply our method to deep unsupervised domain adaptation where the mini-batchbased implementation is required.
The main challenge is that some of the samples in the mini-batch may not be matched.
For instance, the categories of some samples in the source mini-batch may not be present in the target mini-batch, and thus these source samples should not be transported/matched.
Inspired by [65] that uses partial OT over the mini-batch data to implement deepJDOT [66], we use our partial KPG-RL-KP model to partially match the mini-batch data in the training of the deep network.
The partial KPG-RL-KP model is modified from Eq.
(13) by replacing G by αC +(1 − α ) G .
As an experimental example, we apply the partial KPG-RL-KP to the unsupervised domain adaptation experiment on the Office-Home dataset [67].
We take the source and target class centers of the same class as a keypoint pair.
The centers are online updated by exponential moving average in training, same as in [68].
We use the pseudo labels of target data to update the target class centers, due to the lack of target labels.
The protocol is the same as that in [65] .
The batch size is set to 65 and the total transport mass ( s in Eq.
(13)) is set to 0.6, which are the same as those in [65].
The results are reported in the following Table A-3.
[Table]
In Table A-
[Table]
ROT [23] is a robust OT method.
m-OT is the direct mini-batch implementation of deepJDOT [66].
m-UOT [69] and m-POT
[65] are respectively unbalanced deepJDOT and partial deepJDOT on mini-batch data.
m-KPG-RL-KP is the direct mini-batch implementation of our KPGRL-KP model.
m-PKPG-RL-KP is the mini-batch implementation of our partial KPG-RL-KP model.
We can see that by partially matching the samples in the mini-batches, m-KPG-RL-KP outperforms m-KPG-RL-KP by a margin of 6.24%.
Our partial KPG-RL-KP (m-PKPG-RL-KP) outperforms partial DeepJDOT (m-POT) by 0.68%, indicating that using partial matching, our approach is effective for unsupervised domain adaptation under mini-batch implementation.
B.4 Additional Details for HDA Experiments
Kernel SVM.
In the kernel SVM, we use the radial basis function kernel k ( x, y ) =
exp( − γ ∥ x − y ∥ 2 ) , where γ is set to the reciprocal of the feature dimension.
We use the scikit-learn packadge of python to implement it by simply running the following codes:
[Equation]
Barycentric mapping.
The barycentric mapping is defined as follows.
Given the transport plan π ∈ Σ m × n and source data point x i 0 , the barycentric mapping [49] is defined as B π ( x i 0 ) =
arg min y (cid:80) n j =1 π i 0 ,j c ( x i 0 , y j ) .
Since c is the squared L 2 -distance in our paper, B π ( x i 0 ) has closedform expression of
[Equation]
B.5 Additional Ablation Studies
Matching accuracy on real data in HDA application.
We compare the matching accuracy of different OT models on Office-31 dataset in Table A-4.
To compute the matching accuracy, for each transported source data point, we find its nearest neighbor among the target data points to construct a matched pair.
If the two points in a pair have the same class labels, the matching is correct, otherwise the matching is incorrect.
The matching accuracy is the ratio of correctly matched pairs.
In Table A-4, we can see that without the guidance of keypoints, the matching accuracy of GW model is less than 3%.
SGW improves the matching accuracy of GW.
Our proposed KPG-RL and KPG-RL-GW models achieve better matching accuracy than SGW.
[Table]
Comparison of different choices for d .
Since R sk and R tl are in the probability simplex, it is reasonable to measure their difference by a distribution divergence/distance.
The widely used distribution divergences/distances include the KL-divergence, JS-divergence, and Wasserstein distance.
The
KL-divergence is not symmetric, so we need to determine the order of inputs.
For the Wasserstein distance, one should define the ground metric first.
A possible strategy is to set the ground metric to 0 if the two keypoints are paired, otherwise 1.
Such a ground metric makes the Wasserstein distance equal to the L 1 -distance.
In this work, d is taken as the JS-divergence.
We compare the performance of different choices of d in the experiment of HDA on Office-31, as in Table A-5.
[Table]
In Table A-5, KL-ST and KL-TS denote the KL-divergence KL ( R sk , R tl ) and
KL ( R tl , R sk ) respectively.
GW is the Gromov-Wasserstein distance between R sk and R tl where the source/target cost is taken as the L 2 -distance of source/target keypoints.
We find that the JS-divergence achieves the best performance, compared with KL-ST, KL-TS, L 1 -distance, L 2 -distance, and Gromov-Wasserstein.
Results of KPG-RL without using the guiding matrix G .
The guiding matrix is the core to impose the relation preservation.
We below show in Table A-
the results for our KPG-RL without using the guiding matrix G , i .
e ., L kpg ( π ) = ⟨ M, π ⟩ F .
[Table]
We can see that without G , both the results of KPG-RL model and KPG-RL-GW model decrease.
This may be because L kpg ( π ) = ⟨ M, π ⟩ F may not well impose the guidance of keypoints, since it does not model the “relation” of each point to keypoints.
Sensitivity to source keypoints.
In the experiments of the paper, the source keypoints are taken as the source class centers.
To study the sensitivity to the location source keypoints, we randomly sample one data point from each class as a keypoint to construct the source keypoints.
We run the experiments with five different samplings for constructing the source keypoints (these five runs are denoted as S1, S2, S3, S4, S5 respectively).
The results are reported in the Table A-7.
We can see that using the class center as the keypoints achieves the best results, compared with randomly sampling one data point per class as the keypoints.
This may be because the class centers are estimated using all the data of each class, and these centers can better represent each class than a randomly sampled data point of each class.
[Table]
We next study the sensitivity to the number of source keypoints, of which the results are reported in
Table A-8.
In this experiment, we randomly sample 3/5/7/9 samples (keypoints) or use all the source samples (keypoints) for each class in the source domain to compute the source class centers, which are paired with labeled target samples for constructing the keypoint pairs.
The results in Table Ashow that as the number of source keypoints increases, the accuracy gradually increases.
The best result is obtained when all source samples are used to compute the class centers.
[Table]
On defining keypoints in other practical applications.
According to the results in Table A-7, the class centers are better to be the keypoints than the randomly selected samples.
For other practical applications, there may not be “class labels” available.
We could first cluster the points and then annotate the points near to the center of the clusters as the keypoints.
Time and memory cost.
We report the memory and time cost of KPG-RL with different sizes of the guiding matrix G in the bottom row of Tables A-9 and A-10 respectively.
For comparisons, we also report the memory and time cost of the Kantorovich Problem (KP).
KP needs to calculate the pair-wised cost matrix C , as in Eq.
KPG-RL calculates the relation score, and then computes the guiding matrix G .
Since we have deduced Sinkhorn’s algorithm for solving KPG-RL, we solve both
KP and KPG-RL using Sinkhorn’s algorithm with ϵ = 0 .
Table A-9 shows that KPG-RL costs a slightly larger memory than KP.
Table A-10 shows the computational time for solving KPG-RL and KP problems.
In the experiment on Office-31, the maximum memory of G is 38M, and the peak memory of the running process is 780M.
[Table] [Table]
Sensitivity to hyper-parameters.
We show the sensitivity of our method to hyper-parameters τ , τ ′ in Table A-11, ϵ in Table A-12, and α in Table A-13.
ϵ is the the coefficient of entropy regularization.
τ and τ ′ are used to define the relation in Eqs.
(7) and (8) in the paper.
We set τ = ρ max i,j { C si,j } and τ ′ =
ρ max i,j { C ti,j } .
We then show the results with varying values of ρ .
It can be observed that the best value of α is 0.4 in this task, and the results are relatively stable when α ranges in [0.2, 0.5].
[Table]
[Table] [Table]