Unified Optimal Transport Framework for Universal Domain Adaptation
Abstract
Universal Domain Adaptation (UniDA) aims to transfer knowledge from a source domain to a target domain without any constraints on label sets.
Since both domains may hold private classes, identifying target common samples for domain alignment is an essential issue in UniDA.
Most existing methods require manually specified or hand-tuned threshold values to detect common samples thus they are hard to extend to more realistic UniDA because of the diverse ratios of common classes.
Moreover, they cannot recognize different categories among target-private samples as these private samples are treated as a whole.
In this paper, we propose to use
Optimal Transport (OT) to handle these issues under a unified framework, namely
UniOT.
First, an OT-based partial alignment with adaptive filling is designed to detect common classes without any predefined threshold values for realistic
UniDA.
It can automatically discover the intrinsic difference between common and private classes based on the statistical information of the assignment matrix obtained from OT.
Second, we propose an OT-based target representation learning that encourages both global discrimination and local consistency of samples to avoid the over-reliance on the source.
Notably, UniOT is the first method with the capability to automatically discover and recognize private categories in the target domain for UniDA.
Accordingly, we introduce a new metric H 3 -score to evaluate the performance in terms of both accuracy of common samples and clustering performance of private ones.
Extensive experiments clearly demonstrate the advantages of UniOT over a wide range of state-of-the-art methods in UniDA.
Introduction
Deep neural networks have boosted performance in extensive computer vision tasks but still struggle to generalize well in cross-domain tasks that source and target domain data are drawn from the different data distributions.
Unsupervised Domain Adaptation (UDA) [28] aims to transfer knowledge from fully labeled source to unlabeled target domain by minimizing the domain gap between source and target.
However, existing UDA methods tackle the domain gap under a strong closed-set assumption that two domains share identical label sets, limiting their applications to real-world scenarios.
Recently, Partial Domain Adaptation (PDA) [4] and Open Set Domain Adaptation (OSDA) [2] are proposed to relax the closed-set assumption, allowing the existence of private classes in the source and target domain respectively.
However, all the above-mentioned settings heavily rely on
[Figure] [Figure]
prior knowledge where the common classes lie in the target domain.
It may be not reasonable and realistic for UDA since the target domain is unsupervised.
To address the above problem, a generalized setting, termed as Universal Domain Adaptation (UniDA) [39], was proposed to allow both domains to own private classes but without knowing prior information, e.g. matched common classes and classes numbers in the target domain.
Detecting common and target-private samples in the target domain is an essential issue in UniDA.
Existing UniDA methods [39, 17, 32] detect common and target-private samples by using some manually specified or hand-tuned threshold values.
Therefore, these methods are not applicable to more realistic UniDA due to the diverse ratios of common categories as shown in Fig. 1(a).
Moreover, most existing UniDA methods treat all target-private samples as a single class and cannot recognize different categories among them, as shown in Fig. 1(b).
This paper is the first to emphasize that
UniDA methods should have the capability to automatically discover and recognize private categories in the target domain.
Essentially, the common class detection and private class discovery problems can be both viewed as distribution transportation problems.
Therefore, these issues can be formulated within the framework of Optimal Transport (OT), which is a promising optimization problem to seek an efficient solution for transporting one distribution to another.
Even though many OT-based methods [11, 24, 38, 13] have been proposed for Unsupervised Domain Adaptation, most of them consider cross-domain sample-to-sample mapping under the closed-set condition and are not specialized in UniDA problem with unaligned label sets.
Specifically, OT encourages a global mapping to mine domain statistics property for discovering intrinsic differences among common and target private samples.
Additionally, the OT constraints can also avoid degenerate solutions in clustering representation problem [1, 6].
Inspired by this, OT can be a proper formulation with respect to inter-domain common class detection and intra-domain private class discovery.
In this paper, we propose a unified optimal transport framework, namely UniOT, to solve Universal
Domain Adaptation from the perspectives of partial alignment for common class detection and target representation learning for private class discovery.
With regards to common class detection, we propose an OT-based partial alignment to detect common samples in the target domain and develop an adaptive filling method to handle the diverse ratios of common categories.
Without any predefined threshold values, it can automatically discover the intrinsic difference between common and private classes based on the global statistical information of the assignment matrix obtained from OT.
For private class discovery, we propose an OT-based target representation learning that encourages both global discrimination and local consistency of samples to avoid the over-reliance on source supervision.
In addition, UniOT has the capability to automatically discover and recognize categories in the target domain, benefiting from the representation learning with OT.
We believe a competitive UniDA method should achieve high classification accuracy for the common class and also learn a discriminative representation for the target private class.
However, existing methods do not provide any quantitative metric to evaluate the target representation performance for those unknown samples.
For target-private class discovery purpose, we introduce a new evaluation metric,
H 3 -score, considering not only common class accuracy but also the clustering performance of private classes.
Our contributions are summarized as follows: (1) We propose UniOT to handle two essential issues in UniDA, including common class detection and private class discovery.
To our best knowledge, this is the first attempt to jointly consider common class detection and private class discovery in a unified framework by optimal transport.
(2) We propose an OT-based partial alignment with adaptive filling for common class detection without any predefined threshold values.
It can automatically adapt to more realistic UniDA scenarios, where the ratios of common classes are diverse as shown in Fig. 1(a).
(3) We design an OT-based representation learning technique for private class discovery, considering both global discrimination of clusters and local consistency of samples.
Unlike most existing methods that treat all target-private samples as a whole, our UniOT can automatically discover and recognize private categories in the target domain.
Related Work
Universal Domain Adaptation As a more generalized UDA setting, UniDA [39] is more challenging and realistic since the prior information of categories is unknown.
UAN [39],CMU [17] and TNT [8] designed sample-level uncertainty criteria to measure domain transferability.
Samples with lower uncertainty are encouraged for adversarial adaptation with higher weights.
Most UniDA methods detect common samples with the sample-level criteria, which requires some manually specified and hand-tuned threshold values.
Moreover, over-reliance on source supervision under category neglects discriminative representation in the target domain.
DANCE [32] proposed neighborhood clustering as a self-supervised technique to learn features useful for discriminating “unknown” categories.
DCC [23] enumerated cluster numbers of the target domain to obtain optimal crossdomain consensus clusters as common classes, but the consensus clusters are not robust enough due to the hard assignment of K-Means [26].
MATHS [7] detected the existence of the target private class by Hartigan’s dip test in the first stage, then trained hybrid prototypes by learning from K-Means assignment where a fixed hyper-parameter for K-Means was adopted.
In this paper, we use OT to handle both common sample detection and target representation learning under a unified framework.
It is worth noting that our method is adaptive for various unbalanced compositions of common and private classes, which does not require any predefined threshold.
In addition, our method encourages both the global discrimination and the local consistency of samples for target representation learning.
Optimal Transport in Domain Adaptation Optimal transport (OT) was first proposed by Kantorovich [22] to obtain an efficient solution for moving one distribution of mass to another.
Interior point methods can handle OT problem with computational complexity at least O ( d 3 log ( d )) .
Cuturi [10] first proposed to use Sinkorn’s algorithm [34] to compute an approximate transport coupling with an entropic regularization.
This method is lightspeed and can handle large-scale problems efficiently.
Later, DeepJDOT [11] used OT to address the domain adaptation problems to pair samples in the source and target domain.
Accordingly, a new classifier is trained on the transported source data representation.
Since classical OT does not allow partial displacement in transport plan, unbalanced optimal transport (UOT) is proposed to relax equality constraint by replacing the hard marginal constraints of OT with soft penalties using Kullback-Leibler (KL) divergence [16], which can also effectively be solved by generalized Sinkhorn’s algorithm [9].
JUMBOT [13] proposed a minibatch strategy coupled with unbalanced optimal transport, which can yield more robust behavior for handling minibatch UDA and PDA problem.
Also, TS-POT [27] addressed the partial mapping problem in UDA and PDA with partial OT [14].
However, these methods consider a cross-domain sample-to-sample mapping under the closed-set condition or label-set prior, and are not suitable for the unknown category gap problem in UniDA.
Besides, a sample-to-sample mapping neglects the global consensus of two domains and intrinsic differences among common and target private samples.
In this paper, we propose an inter-domain partial alignment based on UOT for common class detection and intra-domain representation learning based on OT for private class discovery, where a sample-to-prototype mapping is designed to encourage a more global-aware assignment.
Clustering for
Representation Learning
Deep clustering has raised increasing attention in deep learning community due to its capability of representation learning for discriminative clusters.
DeepCluster [5] proposed an end-to-end framework that iteratively obtains clustering assignment by
K-Means [26] and updates feature representation by the assignment, while PICA [21] simultaneously learns both feature representation and clustering assignment.
A two-step approach was proposed by SCAN [36], which firstly learns a better initial representation, then implements an end-to-end clustering learning in the next stage.
These deep clustering methods assume the number of clusters is
[Figure]
H
L
P N P N
P N P N
AdaptiveFilling
Common Class Detection
[Figure]
[Figure] [Figure]
… SourcePrototypes
[Figure]
by argmax … known beforehand.
Self-labelling [1] and SwAV [6] formulated the clustering assignment problem into an OT problem for better feature representation learning.
Notably, the number of clusters does not rely on the ground-truth categories, benefiting from the equality constraints in OT.
Therefore,
OT-based representation learning is more suitable for UniDA, where the number of private class is unknown.
However, existing OT-based representation learning methods [1, 6] consider global discrimination of clusters but ignore local consistency of samples, limiting direct applications in
UniDA.
In this paper, we develop an OT-based target representation learning technique for both better global and local structures in UniDA.
Methodology
In Universal Domain Adaptation (UniDA), there is a labeled source domain D s =
{ x si , y si } n s i
=1 and an unlabeled target domain D t
= { x t i } n t
i
=1 .
We use C s and C t to denote the label set of source and target domain respectively.
Denote C =
C s ∩ C t as the common label set shared by both domains.
Let C s =
C s \ C and C t =
C t \ C represent label sets of source private and target private, respectively.
UniDA aims to classify target samples into |C| + 1 classes, where target-private samples are treated as unknown class uniformly.
UniDA aims to transfer knowledge from source to target under domain gap and category gap (i.e., different label sets among source and target domain), which makes it challenging to align common samples and reject private samples.
Moreover, over-reliance on common knowledge extracted from the source domain neglects the target intrinsic structure.
Therefore, in this paper, we propose a unified optimal transport framework to address UniDA problem from two perspectives, i.e., common class detection and private class discovery, as shown in Fig.
3.1 Preliminary
Optimal Transport is a constrained optimization problem that seeks an efficient solution of transporting one distribution of mass to another.
We now briefly recall the well-known optimal transport formulation.
Let Σ r := { x ∈ R r + | x ⊤ 1 r = 1 } denote the probability simplex, where 1 d is a d -dimensional vector of ones.
Given two simplex vectors α ∈ Σ r and β ∈ Σ c , we can write the transport polytope of α and β as follows:
[Equation]
The transport polytope U ( α , β ) can also be interpreted as a set of all possible joint probabilities of ( X, Y ) , where X and Y are two d -dimensional random variables with marginal distribution α and β , respectively.
Following SwAV [6], given a similarity matrix M ∈ R r × c instead of distance matrix in [10], the coupling matrix (or joint probability) Q ∗ mapping α to β can be quantified by optimizing the following maximization problem:
[Equation]
where ε > 0 and H ( Q )
=
− (cid:80) ij Q ij log Q ij is the entropic regularization term.
The optimal Q ∗ has been shown to be unique with the form Q ∗ =
Diag ( u ) exp ( M /ε )
Diag ( v ) , where u and v can be solved by Sinkhorn’s Algorithm [10].
Since Q ∗ satisfies the mass constraint (1) U ( α , β ) strictly, the objective OT ε ( M , α , β ) is not suitable for partial displacement problem.
In light of this, Unbalanced OT is proposed to relax the conservation of marginal constraints by allowing the system to use soft penalties [9].
The unbalanced
OT is formulated as
[Equation]
where D KL is Kullback-Leibler Divergence.
Then the optimization problem (3) can be solved by generalized Sinkhorn’s algorithm [9].
3.2 Inter-domain Partial Alignment for Common Class Detection
To extract common knowledge, we propose an Unbalanced OT-based Common Class Detection (CCD) method, which considers a partial mapping problem between target samples and source prototypes from the perspective of cross domain.
Note that our method uses prototypes to encourage a more global-aware alignment, which discovers intrinsic differences among common and target-private samples by exploiting the statistical information of the assignment matrix obtained from the optimal transport.
Since UniDA allows private classes in both source and target domains, partial alignment should be considered to avoid misalignment between target-private samples and source classes.
Hence we formulate the problem mapping target features to the source prototypes C s =
[ c s 1 , · · · , c s |C s | ] ⊤ with unbalanced OT objective to obtain the optimal assignment matrix Q st , i.e.
[Equation]
(4) where S st = Z t C ⊤ s refers to cosine similarity between target samples and source prototypes, and the mini-batch target ℓ 2 -normalized features Z t =
[ z t 1 , · · · , z tB ] ⊤ with batch-size B are extracted by feature extractor f , i.e. z ti = f
( x ti ) /
∥
f ( x ti ) ∥ 2 .
Here we use the same prototype-based classifier in [32] to learn source prototypes.
Note that the private samples will be assigned with relatively low weights in Q st .
Based on this observation, our CCD selects target samples with top confidence as common classes.
Firstly, the assignment matrix is normalized as ¯ Q st = Q st / (cid:80) Q st .
Then we discover statistical property from normalized ¯ Q st and generate scores from the statistical perspectives of both target samples and source prototypes.
For the i -th row in ¯ Q st , it can be seen as a vanilla prediction probability vector and we can get pseudo-label ˆ y ti by argmax operation.
And we assign target samples confidence score w ti with the maximum value of the i -th row in ¯ Q st , i.e.
[Equation]
A higher score w t i means z t
i is relatively closer to a source prototype than any other samples and is more likely from a common class.
Meanwhile, to select target common samples with top confidence, we also evaluate source prototypes confidence score w sj with the sum of the j -th column, i.e.
[Equation]
Analogously, a higher score w sj means c sj is more likely a common source prototype, which is assigned to target samples more frequently.
Then the common samples are detected by statistics mean, i.e.
[Equation]
where δ i = 1 indicates the sample x ti is detected as common sample with top confidence, which can be assigned with pseudo label ˆ y ti by argmax operation.
We can use pseudo labels of selected target samples to compute inter-domain common class detection loss by standard cross-entropy loss, i.e.
[Equation]
To ensure that statistics mean can discover the intrinsic difference between common and private classes, we need to guarantee that the input features for partial mapping should be sampled from the target domain distribution with enough sampling density.
Therefore, we implement a FIFO memory queue [19] to save previous target features for filling mini-batch features, which avoids the limitation led by the lack of statistical property for target samples.
Adaptive filling for unbalanced proportion of positive and negative.
Unfortunately, using statistics mean as a threshold in Eq.
(7) may misclassify some common samples as private class in some extremely unbalanced cases, such as there are few private samples indeed.
Essentially, the assignment weight is determined by the relative distance between samples and prototypes, and private samples with low similarity set negative examples for the others.
To automatically detect target samples, we design an adaptive filling mechanism for positive and negative unbalanced proportions.
Our strategy is to fill the gap of unbalanced proportion to be balanced like Fig.
2 depicted.
Firstly, samples with high similarity larger than a rough boundary γ are defined as positive and the others are negative.
Then we implement adaptive filling to be balanced when the proportion of positive or negative samples exceeds 50% .
For negative filling, we synthesize fake private feature z ti by mixing up target feature z ti and its farthest source prototypes evenly, i.e. ˆ z ni = 12 (cid:0)
z
[Equation]
c sk ⊤ ) (cid:1) .
For positive filling, we first do the CCD without adaptive filling by Eq.(7) to obtain confident features and then reuse these filtered confident features for positive filling.
Note that the filling samples are randomly sampled from mix-up negatives or CCD positives and the filling size is the gap between unfilled positive and negative.
After adaptive filling for balanced proportion, our CCD is adaptive for detecting confident common samples.
Adaptive update for marginal probability vector.
The marginal probability vector β in Objective (4) can be interpreted as a weight budget for source prototypes to be mapped with target samples.
However, the source domain may own source private class, which is unreasonable to set the weight budget equally for each source prototype, i.e. β = 1 |C s | 1 |C s | .
Therefore, we use the ¯ Q st obtained in the last iteration to update β with the moving average for adapting to the real source distribution , i.e.
[Equation]
where (cid:101) β ( t ) is the sum of column in ¯ Q st solved in the β ( t ) case.
3.3 Intra-domain Representation Learning for Private Class Discovery
To exploit target global and local structure and feature representation, we propose an OT-based Private
Class Discovery (PCD) clustering technique that considers a full mapping problem between target sample features and target prototypes.
Especially, our PCD avoids reliance on the number of target classes and encourages global discrimination of clusters as well as local consistency of samples.
To solve this, we pre-define K learnable target prototypes C t and initialize them randomly, where larger K can be determined by the larger number of target samples.
Then we assign target features ¯
Z t with the prototypes C t by solving the optimal transport optimization problem:
[Equation]
where S tt = ¯ Z t C ⊤ t refers to cosine similarity between target samples and target prototypes, and ¯ Z t is the concatenation of the mini-batch target features Z t and their nearest neighbor features.
The marginal constraint of uniform distribution enforces that on average each prototype is selected at least 2 B/K times in the batch.
The solution Q tt satisfies the constraint strictly, i.e. the sum of each row equals to 1 /K .
To obtain prototype assignment for each target sample, the soft pseudo-label matrix ˜ Q tt =
Q tt × B ensures the i -th row q tti is a probability vector.
To encourage global discrimination of clusters, the learnable prototypes C t and the feature extractor f are optimized by minimizing L global , i.e.
[Equation]
where the cross-entropy loss ℓ ( q i , z i ) is presented as
[Equation]
where p i,k =
[Equation]
To encourage local consistency of samples, we swap prediction between anchor feature z ti and its nearest neighbor feature ˜
z
ti retrieved from the memory queue.
Note that instead of swapped prediction between the different views of anchor image in SwAV [6], our swapped L local is based on anchor-neighbor swapping, i.e.
[Equation]
Eventually, to encourage both global discrimination of clusters and local consistency of samples,
our
L PCD is presented as
[Equation]
Similar to Sec.3.2, since OT encourages a more deliberative mapping under marginal constraint, we also need to fill input features with the previous target features in memory queue before solving the optimization problem (10).
3.4 A Unified Framework for UniDA by Optimal Transport
In UniDA, the partial alignment for common class detection and representation learning for private class discovery are based on UOT and OT models, respectively.
Therefore, we can summarize the above two models in a unified framework, i.e. UniOT.
Considering cross-entropy loss L cls on source samples, Common Class Discovery loss L CCD and Private Class Discovery loss L PCD , our overall objective can be expressed as
[Equation]
In the testing phase, only feature extractor f and prototype-based classifier C s are preserved.
We solve the optimization problem (4) for all the concatenated target features and adaptive filled features.
Then compute w tj in Eq.
5 for each sample.
For those samples who satisfy w tj ≥ 1 n are assigned with the nearest source class, where n is the sum of the total size of target samples n t and adaptive filling size.
Otherwise, the samples are marked as unknown .
Experiments 4.1 Experimental Settings
Dataset.
Our method will be validated in 4 popular datasets in Domain Adaptation.
Office [31] contains 31 categories and about 4K images in 3 domains: Amazon( A ), DSLR( D ) and Webcam( W ).
Office-Home [37] contains 65 categories and about 15K images in 4 domains: Artistic images( Ar ),
Clip-Art images( Cl ), Product images( Pr ) and Real-World images( Rw ).
VisDA [30] is a large dataset which constains 12 categories in source domain with 15K synthetic images and target domain with 5K real-world images.
DomainNet [29] is the largest domain adaptation dataset which containscategories and about 0.6 million in 6 domains but we only use 3 domain: Painting ( P ), Real ( R ), and
Sketch ( S ) like [17].
Note that we use A 2 W to denote that transfer task from Amazon to DSLR.
We follow the dataset split in [17] to conduct experiments.
Evaluation metric.
For UniDA, the trade-off between the accuracy of common and private classes is important in evaluating performance.
Thus, we use the H-score [17] to calculate the harmonic mean of the instance accuracy on common class a C and accuracy on the single private class a ¯ C t .
However,
H-score treats all private samples as a single class.
For target-private class discovery purposes, we further introduce H 3 -score, as the harmonic mean of accuracy on common class, accuracy on unknown class and Normalized Mutual Information (NMI) for target-private clusters, i.e.
[Equation]
16) where NMI is a well-known measure for the quality of clustering, and NMI is obtained by K-Means with the ground truth of the number of private classes in the inference stage.
4.2 Experimental details
Our implementation of optimal transport solver is based on POT [15].
We employ pretrained ResNet-50 [20] on ImageNet [12] as our initial backbone.
The feature extractor consists of backbone and projection layers which are the same as [6] but BatchNorm layer is removed.
We adopt the same optimizer details as [23].
The batch size is set to 36 for both source and target domains.
For all experiments, the initial learning rate is set to 1 × 10 − 2 for all new layers and 1 × 10 − 3 for pretrained backbone.
The total training steps are set to be 10K for all datasets.
For the pre-defined number of target prototypes, a larger size of target domain indicates a larger K .
Therefore, we empirically set
K = 50 for Office, K = 150 for Office-Home, K = 500 for VisDA, K = 1000 for DomainNet.
We default γ = 0 .
7 , µ = 0 .
7 , τ = 0 .
1 , ε = 0 .
01 , κ = 0 .
5 and λ = 0 .
1 for all datasets.
We set the size of memory queue 2K for Office and Office-Home, 10K for VisDA and DomainNet.
All experiments are implemented on a GPU of NVIDIA TITAN V with 12GB.
Each experiment takes about 2.5 hours.
Our code is available at: https://github.com/changwxx/UniOT-for-UniDA .
4.3 Results
Comparison with state-of-the-arts.
Tab. 1 and 2 show the H-score results for Office, Office-Home,
VisDA and DomainNet.
Our UniOT achieves the best performance on all benchmarks.
H-score on non-UniDA methods are reported from [17], and UniOT methods are reported from original papers accordingly, except for DANCE [32] since they did not report H-score and we reproduce it with their released code for fair comparison, marked as ‡ .
In particular, with respect to H 3 -score in Tab. 3, our UniOT surpasses other methods by 7% and 9% in Office and Office-Home datasets respectively, which demonstrates that our UniOT achieves balanced performance for both common class detection and private class discovery.
[Table]
[Table] [Table]
Evaluation of the effectiveness of the proposed CCD and PCD.
To evaluate the contribution of
L CCD , L global and L local , we train the model with different combination of each component.
As shown in Tab.4, the combination of L global and L local , i
[Table]
e. L PCD , brings significant contribution since target representation is crucial for distinguishing common and private samples.
Especially,
L global contributes more than L local , which demonstrates the benefit of global discrimination of clusters for representation learning.
Besides, we also conduct experiments for CCD without adaptive filling and denote the loss as L † CCD , which verifies the effectiveness of adaptive filling design.
[Table]
Robustness in realistic UniDA.
We compare the performance of our UniOT with DCC, DANCE and UAN under different categories split.
In this analysis, we perform on A2W of Office with a fixed common label set C and target-private label set C t , but the different size of source-private label set
C s .
We set 10 categories for C , 11 categories for C t and vary the number of C s .
Fig.
3(a) shows that our UniOT always outperforms other methods.
Besides, UniOT is not sensitive to the variation of source-private classes while other methods are not stable enough.
Additionally, we also analyze the behavior of UniOT under the different size of target-private label set C t .
We also perform on A2W of Office with a fixed common label set C for 10 categories and source-private label set C s forcategories.
Fig. 3(b) shows that our UniOT still performs better, while the other methods present sensitivity to the number of target-private classes.
Therefore, UniOT is more robust to the variation of target-private classes.
Feature visualization for comparison with existing UniDA methods.
We use t-SNE [35] to visualize the learned target features for Rw2Pr of Office-Home.
As shown in Fig.
4, the common samples are colored in black and the private samples are colored with other non-black colors by their ground-truth classes.
Fig. 4(b) shows that UAN cannot recognize different categories among target-private samples since they treat all target-private samples as a single class.
Especially, our
UniOT discovers the global and local structure of target-private samples.
Fig. 4(d) validates that
UniOT learns a better target representation with global discrimination and local consistency of samples compared to DANCE shown in Fig. 4(c).
[Figure] [Figure]
Conclusion
In this paper, we have proposed to use Optimal Transport to handle common class detection and private class discovery for UniDA under a unified framework, namely UniOT.
More precisely, an
OT-based cross-domain partial alignment was designed to detect common class with an adaptive filling strategy to handle the diverse UniDA settings.
In addition, we have proposed an OT-based target representation learning for private class discovery, which encourages both global discrimination and local consistency of samples.
Experimental results on four benchmark datasets have validated the superiority of our UniOT over a wide range of state-of-the-art methods.
In the future, we will modify memory model for more efficient features filling in OT.
Acknowledgement
This work was supported by the Shanghai Sailing Program (21YF1429400, 22YF1428800), Shanghai
Local College Capacity Building Program (23010503100), and Shanghai Frontiers Science Center of
Human-centered Artificial Intelligence (ShangHAI).