Unknown-Aware Domain Adversarial Learning for Open-Set Domain Adaptation
Abstract
Open-Set Domain Adaptation (OSDA) assumes that a target domain contains unknown classes, which are not discovered in a source domain.
Existing domain adversarial learning methods are not suitable for OSDA because distribution matching with unknown classes leads to negative transfer.
Previous OSDA methods have focused on matching the source and the target distribution by only utilizing known classes.
However, this known -only matching may fail to learn the target- unknown feature space.
Therefore, we propose Unknown-Aware Domain Adversarial Learning (UADAL), which aligns the source and the target- known distribution while simultaneously segregating the target- unknown distribution in the feature alignment procedure.
We provide theoretical analyses on the optimized state of the proposed unknown-aware feature alignment, so we can guarantee both alignment and segregation theoretically.
Empirically, we evaluate UADAL on the benchmark datasets, which shows that UADAL outperforms other methods with better feature alignments by reporting state-of-the-art performances † .
Introduction
Unsupervised Domain Adaptation (UDA) means leveraging knowledge from a labeled source domain to an unlabeled target domain [4, 3, 2, 14, 8].
This adaptation implicitly assumes the source and the target data distributions, where each distribution is likely to be drawn from different distributions, i e., domain shift (see Figure 1a).
Researchers have approached the modeling of two distributions by statistical matching [20, 27, 15, 16] or domain adversarial learning [7, 28, 18].
Among them, domain adversarial learning has been successful in matching between the source and the target distributions via feature alignment, so the model can accomplish the domain-invariant representations.
There is another dynamic aspect of the source and the target distribution.
In a realistic scenario, these distributions may expand a class set, which is called unknown classes.
This expansion creates a ﬁeld of Open-Set Domain Adaptation (OSDA) [23, 13].
Existing adversarial learning methods of UDA have limitations to solve OSDA because matching the source and the target distribution with the unknown classes may lead to the negative transfer [6] due to the class set mismatch (see Figure 1b).
Previous OSDA methods focused on matching between the source and the target domain only embedded in the known class set via domain adversarial learning [23, 13].
However, this knownonly feature alignment may fail to learn the target-unknown feature space because of no alignment
[Figure]
signal from the target-unknown instances.
Therefore, a classiﬁer is not able to learn a clear decision boundary for unknown classes because the target-unknown instances are not segregated enough in the aligned feature space (see Figure 1c).
On the other hand, some OSDA methods propose to learn intrinsic target structures by utilizing self-supervised learning without distribution matching [24, 12].
However, this weakens the model performance under large domain shifts.
Therefore, in order to robustly solve OSDA, distribution matching via the domain adversarial learning is required, and the class set mismatch should be resolved to prevent the negative transfer, simultaneously.
This paper proposes a new domain adversarial learning for OSDA, called Unknown-Aware Domain
Adversarial Learning (UADAL).
Speciﬁcally, we aim at enforcing the target-unknown features to move apart from the source and the target-known features, while aligning the source and the target-known features.
We call these alignments as the segregation of the target-unknown features (gray-colored arrow in Figure 1d).
Although this distribution segregation is an essential part of OSDA, the segregation has been only implicitly modeled because of its identiﬁcation on the target-unknown instances.
UADAL is the ﬁrst explicit mechanism to simultaneously align and segregate the three sets (source, target-known, and target-unknown instances) via the domain adversarial learning.
Therefore, the proposed unknown-aware feature alignment enables a classiﬁer to learn a clear decision boundary on both known and unknown class in the feature space.
UADAL consists of three novel mechanisms.
First, we propose a new domain discrimination loss to engage the target-unknown information.
Second, we formulate a sequential optimization to enable the unknown-aware feature alignment, which is suitable for OSDA.
We demonstrate that the optimized state of the proposed alignment is theoretically guaranteed.
Third, we also suggest a posterior inference to effectively recognize target- known / unknown information without any thresholding.
Preliminary 2.1 Open-Set Domain Adaptation
This section provides a formal deﬁnition of OSDA.
The fundamental properties are two folds: 1) the different data distributions of a source domain, p s ( x, y ) , and a target domain, p t ( x, y ) ; and 2) the additional classes of the target domain, which were not observed in the source domain.
Speciﬁcally, we deﬁne the source and the target datasets as χ s = { ( x is , y is ) } n s i =1 and χ t = { ( x it , y it ) } n t i =1 , respectively.
y it is not available at training under the UDA setting.
Additionally, we designate C s to be the source class set (a.k.a.
a shared- known class), and C t to be the target class set, i.e., y it ∈ C t .
OSDA dictates
C s ⊂ C t [23].
C t \ C s is called unknown classes.
In spite that there can be multiple unknown classes, we consolidate C t \ C s as y unk to be a single unknown class, due to no knowledge on C t \ C s .
The learning objectives for OSDA become both 1) the optimal class classiﬁcation in χ t if a target instance belongs to C s , and 2) the optimal unknown classiﬁcation if a target instance belongs to C t \C s .
This objective is formulated as follows, with a classiﬁer f and the cross-entropy loss function L ce ,
[Equation]
2.2 Adversarial Domain Adaptation
We ﬁrst step back from OSDA to Closed-set DA (CDA) with C t \ C s = φ , where the domain adversarial learning is widely used.
Domain-Adversarial Neural Network (DANN)
[7] proposes,
[Equation]
[ L ce ( f ( G ( x s )) , y s )] − L d } .
[Figure]
This objective assumes G ( x ) to be a feature extractor that learns the domain-invariant features, enabled by the minimax game with a domain discriminator D , with respect to L d , as below.
[Equation]
The adversarial framework adapts G ( x ) toward the indistinguishable feature distributions between the source and the target domain by the minimax game on D ( G ( x ))
=
[ D s ( G ( x )) , D t ( G ( x ))] .
Here,
D is set up to have a two-dimensional output to indicate either source or target domain, denoted as
D s and D t , respectively.
Given these binarized outputs of D , this formulation is not appropriate to differentiate the target-known and the target-unknown features in OSDA.
DANN enforces to include the target- unknown features in the distribution matching, which leads to performance degradation by negative transfer [6].
Figure 1b represents this undesired alignment of unknown classes.
Separate To Adapt (STA) [13] proposes a weighted domain adversarial learning, to resolve the negative transfer by utilizing a weighting scheme.
Therefore, STA modiﬁes L d as follows,
[Equation]
where w x represents a probability of an instance x belonging to the shared-known classes.
L STA d enables the domain adversarial learning to align the features from the source and the most likely target-known instances with the high weights.
However, the feature extractor, G , is not able to move apart the target-unknown instances due to lower weights, i.e., no alignment signals (see Figure 1c).
STA modeled this known-only feature alignment by assuming the unknown feature would be implicitly learned through the estimation on w x .
Besides learning w x , the target-unknown instances cannot contribute to the training of STA because their loss contribution will be limited by lower w x , in the second term of Eq.
Therefore, STA does not take information from the target-unknowns, and later Section 4.3 empirically shows that the target-unknown segregation of STA is limited.
2.3 Comparison to Recent OSDA Research
In terms of the domain adversarial learning, in addition to STA, OSBP [23] utilizes a classiﬁer to predict the target instances to the pre-determined threshold, and trains the feature extractor to deceive the classiﬁer for aligning to known classes or rejecting as unknown class.
However, their recognition on unknown class only depends on the threshold value, without considering the data instances.
PGL [19] introduces progressive graph-based learning to regularize the class-speciﬁc manifold, while jointly optimizing the feature alignment via domain adversarial learning.
However, their adversarial loss includes all instances of the target domain, which is critically weakened by the negative transfer.
On the other hand, self-supervised learning approaches have been proposed recently, in order to exploit the intrinsic structure of the target domain [5, 24, 12].
However, these approaches do not have any feature alignments between the source and the target domain, which leads to performance degradation under signiﬁcant domain shifts.
There is also a notable work, OSLPP [31] optimizing projection matrix toward a common subspace to class-wisely align the source and the target domain.
However, the optimization requires pair-wise distance calculation, which results in a growing complexity of
O ( n 2 ) by the n data instances.
It could be limited to the large-scale domain.
We provide detailed comparisons and comprehensive literature reviews in Appendix A.
Methodology 3.1 Overview of UADAL
Figure 2a illustrates the neural network compositions and the information ﬂow.
Our model consists of four networks: 1) a feature extractor, G ; 2) a domain discriminator, D ; 3) an open-set recognizer, E ; and 4) a class classiﬁer network, C .
The networks G , D , E , and C are parameterized by θ g , θ d , θ e , and θ c , respectively.
First, we propose a new domain discrimination loss, and we formulate the sequential optimization problem, followed by theoretical analyses.
In order to recognize targetunknown information, we introduce a posterior inference method, followed by open-set classiﬁcation.
3.2 Sequential Optimization Problem for Unknown-Aware Feature Alignment
Model Structure We assume that we identify three domain types, which are 1) the source ( s ), 2) the target-known ( tk ), and 3) and target-unknown ( tu ).
As the previous works such as STA and DANN utilize a two-way domain discriminator with ( D s , D t ), they can treat only s and tk (or t ).
However, a domain discriminator in OSDA should be able to handle tu information for segregation .
Therefore, we designed the discriminator dimension to be three ( D s , D tk , D tu ), as follows,
[Equation]
Given these three dimensions of D , we propose new domain discrimination losses, L sd and L td , as
[Equation]
where w x := p ( known | x ) , named as open-set recognition , is the probability of a target instance, x , belonging to a shared- known class.
We introduce the estimation of this probability, w x , in the section 3.4 later.
From L td in Eq.
(7), our modiﬁed D becomes to be able to discriminate tk and tu , explicitly.
Our new unknown awareness of D allows us to segregate the target-unknown features via the domain adversarial learning framework.
Firstly, we decompose L td ( θ g , θ d ) into L tkd ( θ g , θ d ) and L tud ( θ g , θ d ) ,
[Equation]
where p tk ( x ) :
= p t ( x | known ) , p tu ( x ) :=
p t ( x | unknown ) , λ tk := p ( known ) , and λ tu :=
p ( unknown ) .
The derivation of Eq.
(8) is based on p t ( x ) =
λ tk p tk ( x ) + λ tu p tu ( x ) , which comes from the law of total probability (Details in Appendix B.1.1).
Therefore, this decomposition of L td into L tud and L tkd enables the different treatments on tk and tu feasible.
The three-way discriminator and its utilization in Eq.
(9)-(10) becomes the unique contribution from UADAL.
Optimization Our goal of this unknown-aware domain discrimination is to align the features from the source and the target-known instances while simultaneously segregating the target-unknown features.
To achieve this goal, we propose a new sequential optimization problem w.r.t.
G and D .
Based on the losses, L sd ( θ g , θ d ) , L tkd ( θ g , θ d ) , and L tud ( θ g , θ d ) , we formulate the optimization problem as,
[Equation]
This alternating objective by Eq.
(11) and Eq.
(12) is equivalent to the adversarial domain adaptation models [7, 13] to learn the domain-invariant features.
Unlike the previous works, however, Eq.
(11) represents that we train the domain discriminator, D , to classify an instance as either source, target-known, or target-unknown domain.
In terms of the feature extractor, G , we propose Eq.
(12) to maximize the domain discrimination loss for the source and the target-known domain while minimizing the loss for the target-unknown domain.
From the different signals on L tk d and L tu d , we only treat the adversarial effects on L sd and L tkd .
Meanwhile, minimizing L tud provides for the network
G to learn the discriminative features on tu .
Eventually, G and D align the source and target-known features and segregate the target-unknown features from the source and the target-known features (the optimization details in Figure 2b).
We provide the theoretic analysis of the optimized state of the proposed feature alignment in the next subsection, which is unexplored in the OSDA ﬁeld.
3.3 Theoretic Analysis of Sequential Optimization
This section provides a theoretic discussion on the proposed feature alignment by our sequential optimization, w.r.t.
Eq. (11)-(12).
Since our ultimate goal is training G to learn unknown-aware feature alignments, we treat G as a leader and D as a follower in the game theory.
We ﬁrst optimize
Eq.
(11) to ﬁnd D ∗ with a ﬁxed G .
The optimal D ∗ given the ﬁxed G results in the below output :
[Equation]
where p avg ( z ) =
( p s ( z ) + λ tk p tk ( z ) + λ tu p tu ( z )) / 2 (Details in Appendix B.1.2).
Here, z ∈ Z stands for the feature space from G , i.e., p d ( z ) =
{ G ( x ; θ g ) | x ∼ p
d ( x ) } where d is s , tk , or tu .
Given D ∗ with the optimal parameter θ ∗ d , we optimize G in Eq.
Here, we show that the optimization w.r.t.
G is equivalent to the weighted summation of KL Divergences, by Theorem 3.1.
Theorem 3.1.
(Proof in Appendix B.1.3) Let θ ∗ d be the optimal parameter of D by optimizing Eq.
Then, −L G ( θ g , θ ∗ d ) can be expressed as, with a constant C 0 ,
[Equation]
We note that p s , p tk , and p tu are the feature distribution of each domain, mapped by G , respectively.
Therefore, we need to minimize Eq.
(14) to ﬁnd the optimal G ∗ .
Having observed Theorem 3.1,
Eq.
(14) requires p tu to have a higher deviation from the average distribution, p avg , while matching towards p s ≈ p avg and p tk ≈ p avg .
From this optimization procedure, we obtain the domain-invariant features over the source and the target-known domain, while segregating the target-unknowns, which is formalized as p s ≈ p tk and p tu ↔ { p tk , p s } .
We show that minimizing Eq.
(14) does not lead to the negative inﬁnity by the third term, regardless of the other KL Divergences, by Proposition 3.2.
Proposition 3.2.
(Proof in Appendix B.1.4) D KL ( p tu (cid:107) p avg ) is bounded to log 2 − log λ tu .
This boundness guarantees that the ﬁrst two KL Divergence terms maintain their inﬂuence on optimizing the parameter θ g of G stably while segregating the target-unknown features.
Furthermore, we show Proposition 3.3 to characterize the feature alignment between p s and p tk under Eq.
Proposition 3.3.
(Proof in Appendix B.1.5) Assume that supp ( p s ) ∩ supp ( p tu ) = ∅ and supp ( p tk ) ∩ supp ( p tu ) = ∅ , where supp ( p ) :=
{ z ∈ Z| p ( z ) > 0 } is the support set of p .
Then, the minimization w.r.t.
G , by Eq.
(14), is equivalent to the minimization of the summation on two f -divergences:
[Equation]
where f 1 ( u ) =
u log u (1 − α ) u + α and f 2 ( u ) =
u log u αu +(1 − α ) , with α = λ tk 1+ λ tk .
Therefore, the minimum of Eq.
(14) is achieved if and only if p s = p tk .
Therefore, under the assumption, Proposition 3.3 theoretically guarantees that the source and the target-known feature distributions are aligned, which represents p s = p tk .
From these analyses, we can theoretically guarantee both alignment and segregation from the proposed optimization.
3.4 Open-Set Recognition via Posterior Inference
This section starts from estimating w x = p ( known | x ) for a target instance, x .
First, we utilize the labeled source dataset, χ s , to train the feature extractor G and the open-set recognizer E , as below,
[Equation]
where E ( G ( x )) ∈
R |C s | contains the softmax activation.
We note that the open-set recognizer, E , is a classiﬁer to learn the decision boundary over C s by the source domain.
Given the decision boundary by E , our assumption for open-set recognition includes two statements: for the target instances, 1) the higher entropy is caused by an uncertain classiﬁcation case, and 2) the unknown class may induce the higher entropy because there are no prior training instances in the unknown classes.
Based on two assumptions, the open-set recognizer, E , will provide higher entropy for the target-unknown instances than the target-known instances.
Therefore, we consider the entropy value as the open-set recognition indicator.
We veriﬁed our assumptions empirically in Figure 8a.
While we notice the entropy value from a single instance as the indicator between known and unknown , the open-set recognition should be holistically modeled over the target domain.
Therefore, we model the mixture of two Beta distributions on the normalized target entropy values as below,
[Equation]
where the deﬁnition of λ tk and λ tu is the same as in the previous subsection.
(cid:96) x is the entropy value for the target instance, x , i.e., (cid:96) x
= H ( E ( G ( x ))) with the entropy function H .
The Beta distribution is a perfect ﬁt for this case of the closed interval support of [0 , 1] , which is the normalized range of the entropy.
Thus, we introduce the estimator ˆ w x of w x by Posterior Inference , from ﬁtting the Beta mixture model through the Expectation-Maximization (EM) algorithm (Details in Appendix B.2.1),
[Equation]
where the denominator is from Eq.
Since the ﬁtting process incorporates all target information, the entire dataset is utilized to set a more informative weighting without any thresholds.
Here, λ tk and λ tu are also explicitly estimated by the EM algorithm since they are not given (see Eq.
(19) in
Appendix B.2.1), whereas a threshold hyperparameter is used by the previous researches [19, 31].
3.5 Open-Set Classiﬁcation
Given the proposed feature alignments, we train a classiﬁer, C , to correctly classify the target instance over C s or to reject as unknown .
Note that the classiﬁer, C , is the extended classiﬁer with the dimension of |C s | + 1 to include the unknown class.
Firstly, we construct the source classiﬁcation loss on χ s .
Secondly, for the target domain, we need to learn the decision boundary for the unknown class.
Based on the proposed open-set recognition, ˆ w x , we train the classiﬁer, C , as follows:
[Equation]
where n t (cid:48) is a normalizing constant, and y unk is the one-hot vector for the unknown class.
Furthermore, we incorporate entropy minimization loss for enforcing the unlabeled target instances to be recognized effectively [9, 13, 5], with the entropy loss function, L H ,
[Equation]
3.6 Conditional UADAL
Domain adversarial learning may fail to capture a class-wise discriminative pattern in the features when the data distributions are complex [17].
Therefore, we also experimented with the conditional
UADAL (cUADAL) with the discriminative information between classes.
Inspired by [17], we replace the input information of the domain discriminator, D , from G ( x ) to [ G ( x ) , C ( x )] in Eq.
cUADAL has the conditions on the prediction of the extended classiﬁer, C , which is including the unknown dimension.
Therefore, cUADAL provides more discriminative information than UADAL.
3.7 Training Procedure
We introduce how to update the networks, E , G , D , and C , based on the proposed losses.
Before we start the training, we brieﬂy ﬁt the parameters of the posterior inference to obtain ˆ w x .
This starting stage trains E and G on the source domain with a few iterations to catch the two modalities of the target entropy values, inspired by the previous work [1], as below,
[Equation]
After the initial setup of ˆ w x , we train D , C , E , and G , alternatively.
This alternation is composed of two phases: (i) learning D ; and (ii) learning G , C , and E networks.
The ﬁrst phase of the alternative iterations starts from learning the domain discriminator, D , by the optimization on L D in Eq.
[Equation] [Equation]
[Table] [Table]
[Table] [Table]
The detailed experimental results including the other metrics such as OS ∗ and UNK are in Appendix C.2.
Based on the updated ˆ θ d the parameter θ g , is associated with both optimization loss and classiﬁcation loss.
Therefore, for the second phase, the parameters, θ g and θ c , are optimized as below,
[Equation]
We also keep updating θ e by Eq.
(20), which leads to a better ﬁtting of the posterior inference on the aligned features (empirically shown in Figure 9b).
Algorithm 1 in Appendix B.3.1 enumerates the detailed training procedures of UADAL.
Computational Complexity is asymptotically the same as existing domain adversarial learning methods, which linearly grow by the number of instances O ( n ) .
The additional complexity may come from the posterior inference requiring O ( nk ) ; k is a constant number of EM iterations (the detailed analysis on the computations in Appendix B.3.2 and C.2.8).
Experiments 4.1 Experimental Settings
Datasets We utilized several benchmark datasets.
Ofﬁce-31 [22] consists of three domains: Amazon (A), Webcam (W), and DSLR (D) with 31 classes.
Ofﬁce-Home [30] is a more challenging dataset with four different domains: Artistic (A), Clipart (C), Product (P), and Real-World (R), containingclasses.
VisDA [21] is a large-scale dataset from synthetic images to real one, with 12 classes.
In terms of the class settings, we follow the experimental protocols by [23].
Baselines We compared UADAL and cUADAL with several baselines by choosing the recent works in CDA, OSDA, and Universal DA (UniDA).
For CDA, we choose DANN [7] and CDAN [17] to show that the existing domain adversarial learning for CDA is not suitable for OSDA.
For OSDA, we compare UADAL to OSBP [23], STA [13], PGL [19], ROS [5], and OSLPP [31].
For UniDA, we conduct the OSDA setting in DANCE [24] and DCC [12].
Implementation We commonly utilize three alternative pre-trained backbones: 1) ResNet-
[10], 2) DenseNet-121 [11], and 3) EfﬁcientNet-B0 [26] to show the robustness on backbone network choices, while using VGGNet [25] on VisDA for a fair comparison.
(Details in Appendix C.1).
Metric We utilize HOS, which is a harmonic mean of OS ∗ , and UNK [5].
OS ∗ is a class-wise averaged accuracy over known classes, and UNK is an accuracy only for the unknown class.
HOS is suitable for OSDA because it is higher when performing well in both known and unknown classiﬁcations.
[Table] [Table]
Schemes
4.2 Experimental Results
Quantitative Analysis Table 1 reports the quantitative performances of UADAL, cUADAL, and baselines applied to Ofﬁce-31 and Ofﬁce-Home, with three backbone networks.
UADAL and cUADAL show statistically signiﬁcant improvements compared to all existing baselines in both
Ofﬁce-31 and Ofﬁce-Home.
Moreover, there is a large improvement by UADAL from baselines in
EfﬁcientNet-B0 and DenseNet-121, even with fewer parameters.
In terms of the table on ResNet50, for a fair comparison, UADAL outperforms the most competitive model, OSLPP (HOS=67.0).
Compared to the domain adversarial learning methods, such as STA and OSBP, UADAL signiﬁcantly outperforms in all cases.
UADAL also performs better than other self-supervised learning methods such as ROS and DCC.
These performance increments are commonly observed in all three backbone structures and in two benchmark datasets, which represents the effectiveness of the proposed model.
Table 2 shows the experimental results on VisDA, which is known to have a large domain shift from a synthetic to a real domain.
It demonstrates that UADAL and cUADAL outperform the other baselines ‡ under the signiﬁcant domain shift.
Moreover, the baselines with the feature alignments by the domain adversarial learning, such as STA, OSBP, and PGL, outperform the self-supervised learning models, ROS, DANCE, and DCC, in most cases.
Therefore, these results support that
OSDA essentially needs the feature alignments when there are signiﬁcant domain shifts.
The detailed discussions about the low accuracies of some baselines are in Appendix C.2.2.
In addition, we observe that cUADAL mostly performs better than UADAL, which shows that cUADAL provides more discriminative information both on known and unknown by the prediction of the classiﬁer C .
[Figure]
Qualitative Analysis Figure 3 represents the t-SNE visualizations [29] of the learned features by
ResNet-50 (the details in Appendix C.2.3).
We observe that the target-unknown (red) features from the baselines are not segregated from the source (green) and the target-known (blue) features.
On the contrary, UADAL aligns the features from the source and the target-known instances accurately with clear segregation of the target-unknown features.
In order to investigate the learned feature distributions quantitatively, Figure 4 provides Proxy A -Distance (PAD) between the feature distributions from STA, ROS, and UADAL.
PAD is an empirical measure of distance between domain distributions [8] (Details in Appendix C.2.4), and a higher PAD means clear discrimination between two distributions.
Thus, we calculate PAD between the target-known ( tk ) and the target-unknown ( tu ) feature distributions from the feature extractor, G .
We found that PAD from UADAL is much higher than STA and ROS.
It shows that UADAL explicitly segregates the unknown features.
In order to provide more insights and explanations compared to STA, we provide an analysis of the correlation between the evaluation metric and the distance measure of the feature distributions.
We utilize UNK, and HOS as evaluation metrics, and PAD between tk and tu as the distance measure.
Figure 6 shows the scatter plots of (UNK & PAD) and (HOS & PAD).
The grey arrows mean the corresponding tasks by UADAL and STA.
As shown in the ﬁgure, HOS and UNK have a positive correlation with the distance between the target-unknown ( tu ) and the target-known ( tk ).
In simple as VisDA dataset (Detailed discussions
Appendix C.2.1).
words, better segregation enables better HOS and UNK.
Speciﬁcally, from STA to UADAL on the same task, the distances (PAD on tk / tu ) are increased, and the corresponding performances, UNK and
HOS, are also increased.
It means that UADAL effectively segregates the target feature distributions, and then leads to better performance for OSDA.
In other words, the proposed unknown-aware feature alignment is important to solve the OSDA problem.
From this analysis, our explicit segregation of tu is a key difference from the previous methods.
Therefore, we claimed that this explicit segregation of
UADAL is the essential part of OSDA, and leads to better performances.
Robust on Openness We conduct the openness experiment to show the robustness of the varying number of the known classes.
Figure 5 represents that UADAL performs better than the baselines over all cases.
Our open-set recognition does not need to set any thresholds since we utilize the loss information of the target domain, holistically.
Therefore, UADAL is robust to the degree of openness.
4.3 Ablation Studies
Effectiveness of Unknown-Aware Feature Alignment We conducted ablation studies to investigate whether the unknown-aware feature alignment is essential for the OSDA problem.
Table 3 represents the experimental results where the feature alignment procedure of STA or ROS is replaced by UADAL, maintaining their weighting schemes, in Ofﬁce-31 with ResNet-50.
In terms of each weighting, the UADAL core leads to better performances in both STA and ROS cases.
Surprisingly, when we replace STA’s known-only matching with the unknown-aware feature alignment of UADAL, the performance increases dramatically.
It means that the implicit separation of STA at the classiﬁcation level is limited to segregating the target-unknown features.
Therefore, explicit segregation by UADAL is an essential part to solve the OSDA problem.
Moreover,
Figure 7 shows the empirical convergence of the proposed sequential optimization as PAD value between p s and p tk and between p s and p tu over training, i.e., p s ≈ p tk and p tu ↔ { p tk , p s } .
[Figure]
[Figure]
Superiorities of Open-set Recognition: (Effective Recognition) Figure 8a represents the histogram of the source/target entropy values from E after the early stage of training by Eq.
(20) (sensitivity analysis on the varying number of iterations for the early stage in Appendix C.2.5).
Therefore, it demonstrates that our assumption on two modalities of the entropy values holds, robustly.
Furthermore, the subﬁgure in Figure 8a shows the histogram of the ˆ w x by the posterior inference.
It shows that our open-set recognition , ˆ w x , provides a more clear separation for the target-known (blue) and the target-unknown (red) instances than the entropy values.
Figure 8b represents the histogram of the entropy at the end of the training, which is further split than Figure 8a.
This clear separation implicitly supports that the proposed feature alignment accomplishes both alignment and segregation .
(Informativeness)
Furthermore, we conduct a binary classiﬁcation task by thresholding the weights, ˆ w x , whether it predicts tk or tu correctly.
Figure 9a represents the accuracies over the threshold values when applying the posterior inference or the normalized entropy values themselves, in Ofﬁce-31.
It shows that the posterior inference discriminates tk and tu more correctly than using the entropy value itself.
Additionally, the posterior inference has similar accuracies over the threshold values (0.2, 0.5, and 0.8), which also indicates a clear separation.
Therefore, we conclude that the posterior inference provides a more informative weighting procedure than the entropy value itself.
(Comparison to Thresholding) Figure 9b shows that UADAL with the posterior inference outperforms all cases of thresholding the entropy values.
It means that the posterior inference is free to set thresholds.
Moreover, the results demonstrate that UADAL with the posterior inference outperforms the case of using the normalized entropy value, and its frequent updating (solid line) leads to a better performance than only initial updating (dashed line).
It is consistent with the analysis from Figure 8b.
[Table]
in Table 2 in Appendix C.2.7.
(Efﬁciency)
In terms of computation, the posterior inference increases the complexity because we need to ﬁt the mixture model, where the computational complexity is O ( nk ) with the number of samples, n , and the number of ﬁtting iterations, k ( n (cid:29) k ).
As an alternative, we ﬁt the mixture model only by sampling the target instance.
Figure 10 represents the wall-clock time increased by the ﬁtting process and the performances over the sampling ratio (%) for the target domain.
Since the computational complexity is linearly increased by the number of samples, the wall-clock time is also linearly increased by increasing the sampling ratio.
Interestingly, we observed that even when the sampling ratio is small, i.e. 10%, the performances do not decrease, on both Ofﬁce-31 and
Ofﬁce-Home datasets.
The qualitative analysis of this sampling is provided in Appendix C.2.8.
Entropy Minimization Loss
In order to show the effect of the entropy minimization by Eq.
(19), we conduct the ablation study on the Ofﬁce-31 and Ofﬁce-Home.
Table 4 shows that the entropy minimization leads to better performance.
However, UADAL without the loss still performs better than the other baselines (compared with Table 1).
It represents that UADAL learns the feature space appropriately as we intended to suit Open-Set Domain Adaptation.
The properly learned feature space leads to effectively classifying the target instances without entropy minimization.
Conclusion
We propose Unknown-Aware Domain Adversarial Learning (UADAL), which is the ﬁrst approach to explicitly design the segregation of the unknown features in the domain adversarial learning for
OSDA.
We design a new domain discrimination loss and formulate the sequential optimization for the unknown-aware feature alignment.
Empirically, UADAL outperforms all baselines on all benchmark datasets with various backbone networks, by reporting state-of-the-art performances.
Acknowledgments and Disclosure of Funding
This research was supported by Research and Development on Key Supply Network Identiﬁcation,
Threat Analyses, Alternative Recommendation from Natural Language Data (NRF) funded by the
Ministry of Education (2021R1A2C200981612).
Checklist
For all authors... (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contributions and scope?
[Yes] (b) Did you describe the limitations of your work?
[Yes] Details in Appendix D. (c) Did you discuss any potential negative societal impacts of your work?
[Yes] Details in Appendix D. (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[Yes] (b) Did you include complete proofs of all theoretical results?
[Yes] 3.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] We provide code in supplemental material, and URL of data and instructions are provided in Appendix C.1.5.
(b) Did you specify all the training details (e
g., data splits, hyperparameters, how they were chosen)?
[Yes] (c) Did you report error bars (e
g., with respect to the random seed after running experiments multiple times)?
[Yes] (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] The details in Appendix C.1.1. 4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes] (b) Did you mention the license of the assets?
[Yes] (c) Did you include any new assets either in the supplemental material or as a URL?
[Yes] See Appendix C.1.4 and C.1.5.
(d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
[Yes] We mentioned that we use the public benchmark dataset, in Section 4.1.
(e) Did you discuss whether the data you are using/curating contains personally identiﬁable information or offensive content?
[Yes] We use image object recognition datasets, so there are no personaly identiﬁable information or offensive content.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]