Unknown-Aware Domain Adversarial Learning for Open-Set Domain Adaptation
Abstract
Open-Set Domain Adaptation (OSDA) assumes that a target domain containsunknown classes, which are not discovered in a source domain.
Existing domain adversarial learning methods are not suitable for OSDA because distributionmatching with unknown classes leads to the negative transfer.
Previous OSDAmethods have focused on matching the source and the target distribution by onlyutilizing known classes.
However, this known -only matching may fail to learn thetarget- unknown feature space.
Therefore, we propose Unknown-Aware Domain
Adversarial Learning (UADAL), which aligns the source and the targe- knowndistribution while simultaneously segregating the target- unknown distribution inthe feature alignment procedure.
We provide theoretical analyses on the optimizedstate of the proposed unknown-aware feature alignment, so we can guarantee bothalignment and segregation theoretically.
Empirically, we evaluate UADAL on thebenchmark datasets, which shows that UADAL outperforms other methods withbetter feature alignments by reporting the state-of-the-art performances.
Introduction
Unsupervised Domain Adaptation (UDA) means leveraging knowledge from a labeled source domainto an unlabeled target domain [4, 3, 2, 14, 8].
This adaptation implicitly assumes the source and thetarget data distributions, where each distribution is likely to be drawn from different distributions,i e., domain shift (see Figure 1a).
Researchers have approached the modeling of two distributions bystatistical matching [20, 27, 15, 16] or domain adversarial learning [7, 28, 18].
Among them, domainadversarial learning has been successful in matching between the source and the target distributionsvia feature alignment, so the model can accomplish the domain-invariant representations.
There is another dynamic aspect of the source and the target distribution.
In a realistic scenario, thesedistributions may expand a class set, which is called unknown classes.
This expansion creates a ﬁeldof Open-Set Domain Adaptation (OSDA) [23, 13].
Existing adversarial learning methods of UDAhave limitations to solve OSDA because matching the source and the target distribution with theunknown classes may lead to the negative transfer [6] due to the class set mismatch (see Figure 1b).
Previous OSDA methods focused on matching between the source and the target domain only embedded in the known class set via domain adversarial learning [23, 13].
However, this knownonly feature alignment may fail to learn the target-unknown feature space because of no alignment signal from the target-unknown instances.
Therefore, a classiﬁer is not able to learn a clear decisionboundary for unknown classes because the target-unknown instances are not segregated enough inthe aligned feature space (see Figure 1c).
On the other hand, some OSDA methods propose to learnintrinsic target structures by utilizing self-supervised learning without distribution matching [24, 12]
However, this weakens the model performance under large domain shifts.
Therefore, in order to
[Figure]
robustly solve OSDA, distribution matching via the domain adversarial learning is required, and theclass set mismatch should be resolved to prevent the negative transfer, simultaneously.
This paper proposes a new domain adversarial learning for OSDA, called Unknown-Aware Domain
Adversarial Learning (UADAL).
Speciﬁcally, we aim at enforcing the target-unknown featuresto move apart from the source and the target-known features, while aligning the source and thetarget-known features.
We call these alignments as the segregation of the target-unknown features(illustrated as gray-colored arrow in Figure 1d).
Although this distribution segregation is an essentialpart for OSDA because of its identiﬁcation on the target-unknown instances, the segregation hasbeen only implicitly modeled because there was no explicit mechanism to simultaneously align andsegregate three sets: source, target-known, and target-unknown instances.
UADAL is the ﬁrst explicitmechanism in the OSDA ﬁeld to simultaneously align and segregate the three sets via the domainadversarial learning.
Therefore, the proposed unknown-aware feature alignment enables a classiﬁerto learn a clear decision boundary on both known and unknown class in the feature space.
UADAL consists of three novel mechanisms.
First, we propose a new domain discrimination loss toengage the target-unknown information.
Second, we formulate a sequential optimization to enable theunknown-aware feature alignment, which is suitable for OSDA.
We demonstrate that the optimizedstate of the proposed alignment is theoretically guaranteed.
Third, we also suggest a posteriorinference to effectively recognize target- known / unknown information without any thesholding.
Preliminary
2.1 Open-Set Domain Adaptation
This section provides a formal deﬁnition of OSDA.
The fundamental properties are two folds: 1) thedifferent data distributions of a source domain, p s ( x, y ) , and a target domain, p t ( x, y ) ; and 2) the additional classes of the target domain, which were not observed in the source domain.
Speciﬁcally, wedeﬁne the source and the target datasets as χ s = { ( x is , y is ) } n s i =1 and χ t = { ( x it , y it ) } n t i =1 , respectively.y it is not available at training under the UDA setting.
Additionally, we designate C s to be the sourceclass set (a.k.a.
a shared- known class), and C t to be the target class set, i.e., y it ∈ C t .
OSDA dictates
C s ⊂ C t [23].
C t \ C s is called unknown classes.
In spite that there can be multiple unknown classes,we consolidate C t \ C s as y unk to be a single unknown class, due to no knowledge on C t \ C s .
The learning objective for OSDA become both 1) the optimal class classiﬁcation in χ t if a targetinstance belongs to C s , and 2) the optimal unknown classiﬁcation if a target instance belongs to C t \C s .
This objective is formulated as follows, with a classiﬁer f and the cross-entropy loss function L ce ,
[Equation]
2.2 Adversarial Domain Adaptation
We ﬁrst step back from OSDA to Closed-set DA (CDA) with C t \ C s = φ , where the domainadversarial learning is widely used.
Domain-Adversarial Neural Network (DANN)
[7] proposes,
[Equation]
This objective assumes G ( x ) to be a feature extractor that learns the domain-invariant features,enabled by the minimax game with a domain discriminator D , with respect to L d , as below.
[Equation]
[Figure]
The adversarial framework adapts G ( x ) toward the indistinguishable feature distributions betweenthe source and the target domain by the minmax game on D ( G ( x )) =
[ D s ( G ( x )) , D t ( G ( x ))] .
Here,
D is setup to have a two-dimensional output to indicate either source or target domain, denoted as
D s and D t , respectively.
Given these binarized outputs of D , this formulation is not appropriate todifferentiate the target-known and the target-unknown features in OSDA.
DANN enforces to includethe target- unknown features in the distribution matching, which leads to performance degradation bynegative transfer [6].
Figure 1b represents this undesired alignment of unknown classes.
Separate To Adapt (STA) [13] proposes a weighted domain adversarial learning, to resolve thenegative transfer by utilizing a weighting scheme.
Therefore, STA modiﬁes L d as follow,
[Equation]
where w x represents a probability of an instance x belonging to the shared-known classes, or not.
L STA d enables the domain adversarial learning to align the features from the source and the mostlikely target-known instances with the high weights.
However, the feature extractor, G , is not able tomove apart the target-unknown instances due to lower weights, i.e., no alignment signals (see Figure1c).
STA modeled this known-only feature alignment by assuming the unknown feature would beimplicitly learned through the estimation on w x .
Besides learning w x , the target-unknown instancescannot contribute to the training of STA because their loss contribution will be limited by lower w x ,in the second term of Eq.
Therefore, STA does not take information from the target-unknowns,and later Section 4.3 empirically shows that the target-unknown segregation of STA is limited.
2.3 Comparison to Recent OSDA Researches
In terms of the domain adversarial learning, in addition to STA, OSBP [23] utilizes a classiﬁer topredict the target instances to the pre-determined threshold, and trains the feature extractor to deceivethe classiﬁer for aligning to known classes or rejecting as unknown class.
However, their recognitionon unknown class only depends on the threshold value, without considering the data instances.
PGL[19] introduces progressive graph-based learning to regularize the class-speciﬁc manifold, whilejointly optimizing the feature alignment via domain adversarial learning.
However, their adversarialloss includes all instances of the target domain, which is critically weakened by the negative transfer.
On the other hand, self-supervised learning approaches have been proposed recently, in order toexploit the intrinsic structure of the target domain [5, 24, 12].
However, these approaches do nothave any feature alignments between the source and the target domain, which leads to performancedegradation under the signiﬁcant domain shifts.
There is also a notable work, OSLPP [31] optimizingprojection matrix toward a common subspace to class-wisely align the source and the target domain.
However, the optimization requires the pair-wise distance calculation, which results in a growingcomplexity of O ( n 2 ) by the n data instances.
It could be limited to the large-scaled domain.
Weprovide the detailed comparisons and the comprehensive literature reviews in Appendix 1.
Methodology
[Figure]
3.1 Overview of UADAL
[Figure]
Figure 2a illustrates the neural network compositions and the information ﬂow.
Our model consists offour networks: 1) a feature extractor, G ; 2) a domain discriminator, D ; 3) an open-set recognizer, E ; and 4) a class classiﬁer network, C .
The networks E , G , D , and C are parameterized by θ e , θ g , θ d ,and θ c , respectively.
First, we propose a new domain discrimination loss, and we formulate thesequential optimization problem, followed by theoretical analyses.
In order to recognize targetunknown information, we introduce a posterior inference method, followed by open-set classiﬁcation.
3.2 Sequential Optimization Problem for Unknown-Aware Feature Alignment
Model Structure We assume that we identify three domain types, which are 1) the source ( s ), 2) thetarget-known ( tk ), and 3) and target-unknown ( tu ).
As the previous works such as STA and DANNutilize a two-way domain discriminator with ( D s , D t ), they can treat only s and tk (or t ).
However, adomain discriminator in OSDA should be able to handle tu information for segregation .
Therefore,we designed the discriminator dimension to be three ( D s , D tk , D tu ), as follows,
[Equation]
[Equation]
Given these three dimensions of D , we propose new domain discrimination losses, L sd and L td , as
[Equation]
where w x = p ( known | x ) , named as open-set recognition , is the probability of a target instance, x ,belonging to a shared- known class.
We introduce the estimation of this probability, w x , in the nextsection 3.4.
From L td in Eq.
(7), our modiﬁed D is able to discriminate tk and tu , explicitly.
Our new unknown-awareness of D allows us to segregate the target-unknown features via the domainadversarial learning framework.
Firstly, we decompose L td ( θ g , θ d ) into L tkd ( θ g , θ d ) and L tud ( θ g , θ d ) ,
[Equation]
[Equation]
where p tk ( x ) :
= p t ( x | known ) , p tu ( x ) :=
p t ( x | unknown ) , λ tk := p ( known ) , and λ tu :=
p ( unknown ) .
The derivation of Eq.
(8) is based on p t ( x ) =
λ tk p tk ( x ) + λ tu p tu ( x ) , which comesfrom the law of total probability (Details in Appendix 2.1.1).
Therefore, this decomposition of L tdinto L tud and L tkd enables the different treatments on tk and tu feasible.
The three-way discriminatorand its utilization in Eq.
(9)-(10) becomes the unique contribution from UADAL.
Optimization Our goal of this unknown-aware domain discrimination is to align the features from thesource and the target-known instances while simultaneously segregating the target-unknown features.
To achieve this goal, we propose a new sequential optimization problem w.r.t.
G and D .
Based onthe losses, L sd ( θ g , θ d ) , L tkd ( θ g , θ d ) , and L tud ( θ g , θ d ) , we formulate the optimization problem as
[Equation]
This alternating objective by Eq.
(11) and Eq.
(12) is equivalent to the adversarial domain adaptationmodels [7, 13] to learn the domain-invariant features.
Unlike the previous works, however, Eq.(11) represents that we train the domain discriminator, D , to classify an instance as either source,target-known, or target-unknown domain.
In terms of the feature extractor, G , we propose Eq.
(12)to maximize the domain discrimination loss for the source and the target-known domain whileminimizing the loss for the target-unknown domain.
From the different signals on L tud and L tkd , weonly treat the adversarial effects on L sd and L tkd .
Meanwhile, minimizing L tud provides for the network
G to learn the discriminative features on tu .
Eventually, G and D align the source and target-knownfeatures and segregate the target-unknown features from the source and the target-known features(the optimization details in Figure 2b).
We provide the theoretic analysis on the optimized state of theproposed feature alignment in the next subsection, which is unexplored in the OSDA ﬁeld.
Key difference to STA:
STA separates the target-unknown features by training a classiﬁer withknown-only feature alignment, which is implicit separation.
Also, the progressive optimization of
STA is not able to guarantee the separation of tu (and also s ≈ tk ).
With this point, our novelty of
UADAL is the ﬁrst model to propose aligning s and tk and simultaneously segregating tu explicitly ,which is supported by a theoretical guarantee, in the domain adversarial learning framework.
3.3 Theoretic Analysis of Sequential Optimization
This section provides a theoretic discussion on the proposed feature alignment by our sequentialoptimization, w.r.t.
Eq. (11)-(12).
Since our ultimate goal is training G to learn unknown-awarefeature alignments, we treat G as a leader and D as a follower in the game theory.
We ﬁrst optimize
Eq.
(11) to ﬁnd D ∗ with a ﬁxed G .
The optimal D ∗ given the ﬁxed G results in the below output :
[Equation]
where p avg ( z ) =
( p s ( z ) + λ tk p tk ( z ) + λ tu p tu ( z )) / 2 (Details in Appendix 2.1.2).
Here, z ∈ Zstands for the feature space from G , i.e., p d ( z ) =
{ G ( x ; θ g ) | x ∼ p
d ( x ) } where d is s , tk , or tu .
Given D ∗ with the optimal parameter θ ∗ d , we optimize G in Eq.
Here, we show that theoptimization w.r.t.
G is equivalent to the weighted summation of KL Divergences, by Theorem 3.1.
Theorem 3.1.
Let θ ∗ d be the optimal parameter of D by optimizing Eq.
Then, −L G ( θ g , θ ∗ d ) canbe expressed as, with a constant C 0 , (Proof in Appendix 2.1.3)
[Equation]
[Equation]
163 We note that p s , p tk , and p tu are the feature distribution of each domain, mapped by G , respectively.
Therefore, we need to minimize Eq.
(14) to ﬁnd the optimal G ∗ .
Having observed Theorem 3.1,
Eq.
(14) requires p tu to have a higher deviation from the average distribution, p avg , while matchingtowards p s ≈ p avg and p tk ≈ p avg .
From this optimization procedure, we obtain the domain-invariantfeatures over the source and the target-known domain, while segregating the target-unknowns, whichis formalized as p s ≈ p tk and p tu ↔ { p tk , p s } .
We show that minimizing Eq.
(14) does not lead tothe negative inﬁnity by the third term, regardless of the other KL Divergences, by Proposition 3.2.
Proposition 3.2.
(Proof in Appendix 2.1.4) D KL ( p tu (cid:107) p avg ) in Eq.
(14), is bounded to log 2 − log λ tu .
This boundness guarantees that the ﬁrst two KL Divergence terms maintain their inﬂuence onoptimizing the parameter θ g of G stably while segregating the target-unknown features.
Furthermore,we show Proposition 3.3 to characterize the feature alignment between p s and p tk under Eq.
Proposition 3.3.
Assume that supp ( p s ) ∩ supp ( p tu ) = ∅ and supp ( p tk ) ∩ supp ( p tu ) = ∅ , wheresupp ( p ) :
= { z ∈ Z| p ( z ) > 0 } is the support set of p .
Then, the minimization problem w.r.t.
G , Eq.(14), is equivalent to the minimization problem of the summation on two f -divergences:
[Equation]
where f 1 ( u ) =
u log u (1 − α ) u + α and f 2 ( u ) =
u log u αu +(1 − α ) , with α = λ tk 1+ λ tk .
Therefore, theminimum of Eq.
(14) is achieved if and only if p s = p tk (Proof in Appendix 2.1.5).
Therefore, under the assumption, Proposition 3.3 theoretically guarantees that the source and thetarget-known feature distributions are aligned, which represents p s = p tk .
From these theoreticanalyses, we can guarantee both alignment and segregation theoretically.
3.4 Open-Set Recognition via Posterior Inference
This section starts from estimating w x = p ( known | x ) for a target instance, x .
First, we utilize thelabeled source dataset, χ s , to train the feature extractor G and the open-set recognizer E , as below,
[Equation]
where E ( G ( x )) ∈
R |C s | contains the softmax activation.
We note that the open-set recognizer, E , isa classiﬁer to learn the decision boundary over C s by the source domain.
Given the decision boundary by E , our assumption for open-set recognition includes two statements:for the target instances, 1) the higher entropy is caused by an uncertain classiﬁcation case, and 2)the unknown class may induce the higher entropy because there are no prior training instances in theunknown classes.
Based on two assumptions, the open-set recognizer, E , will provide higher entropyfor the target-unknown instances than the target-known instances.
Therefore, we consider the entropyvalue as the open-set recognition indicator.
We veriﬁed our assumptions empirically in the Figure 6a.
While we notice the entropy value from a single instance as the indicator between known and unknown ,the open-set recognition should be holistically modeled over the target domain.
Therefore, we modelthe mixture of two Beta distributions on the normalized target entropy values as below,
[Equation]
where the deﬁnition of λ tk and λ tu is the same as the previous subsection.
(cid:96) x is the entropy value forthe target instance, x , i.e., (cid:96) x
= H ( E ( G ( x ))) with the entropy function H .
The Beta distributionis a perfect ﬁt for this case of the closed interval support of [0 , 1] , which is the normalized range ofthe entropy.
Thus, we introduce the estimator ˆ w x of w x by Posterior Inference , from ﬁtting the Betamixture model through the Expectation-Maximization (EM) algorithm (Details in Appendix 2.2.1),
[Equation]
where the denominator is from Eq.
Since the ﬁtting process incorporates all target information,the entire dataset is utilized to set a more informative weighting without any thresholds.
Here, λ tkand λ tu are also explicitly estimated by the EM algorithm since they are not given (see Eq.
(19) in
Appendix 2.2.1), whereas a threshold hyperparameter is used by the previous researches [19, 31].
3.5 Open-Set Classiﬁcation
Given the proposed feature alignments, we train a classiﬁer, C , to correctly classify the target instanceover C s or to reject as unknown .
Note that the classiﬁer, C , is the extended classiﬁer with thedimension of |C s | + 1 to include the unknown class.
Firstly, we construct the source classiﬁcationloss on χ s .
Secondly, for the target domain, we need to learn the decision boundary for the unknownclass.
Based on the proposed open-set recognition, ˆ w x , we train the classiﬁer, C , as follows:
[Equation]
where n (cid:48) t is a normalizing constant, and y unk
=
[ (cid:126) 0 , 1] is the one-hot vector for the unknown class.
Furthermore, we incorporate entropy minimization loss for enforcing the unlabeled target instancesto be recognized effectively [9, 13, 5], with the entropy loss function, L H ,
[Equation]
3.6 Conditional UADAL
Domain adversarial learning may fail to capture a class-wise discriminative pattern in the featureswhen the data distributions are complex [17].
Therefore, we also experimented the conditional
UADAL (cUADAL) with the discriminative information between classes.
Inspired by [17], wereplace the input information of the domain discriminator, D , from G ( x ) to [ G ( x ) , C ( x )] in Eq.
cUADAL has the conditions on the prediction of the extended classiﬁer, C , which is including theunknown dimension.
Therefore, cUADAL provides more discriminative information than UADAL.
3.7 Training Procedure
We introduce how to update the networks, E , G , D , and C , based on the proposed losses.
Before westart the training, we brieﬂy ﬁt the parameters of the posterior inference to obtain ˆ w x .
This startingstage trains E and G on the source domain with a few iterations to catch the two modalities of thetarget entropy values, inspired by the previous work [1], as below,
[Equation]
After the initial setup of ˆ w x , we train D , C , E , and G , alternatively.
This alternation is composed oftwo phases: (i) learning D ; and (ii) learning G , C , and E networks.
The ﬁrst phase of the alternativeiterations starts from learning the domain discriminator, D , by the optimization on L D in Eq.
[Equation] [Equation]
[Table] [Table]
[Table] [Table]
The detailed results including OS ∗ and UNK are in Appendix 3.2.
Based on the updated ˆ θ d , the parameter, θ g , is associated with both optimization loss and classiﬁcationloss.
Therefore, for the second phase, the parameters, θ g and θ c , are optimized as below,
[Equation]
We also keep updating θ e by Eq.
(20), which leads to a better ﬁtting of the posterior inference on thealigned features (empirically shown in Figure 7b).
Algorithm 1 in Appendix 2.3.1 enumerates thedetailed training procedures of UADAL.
Computational Complexity is asymptotically the same asexisting domain adversarial learning methods, which linearly grows by the number of instances O ( n ) .
The additional complexity may come from the posterior inference requiring O ( nk ) ; k is a constantnumber of EM iterations (the detailed analysis on the computations in Appendix 2.3.2 and 3.2.9).
Experiments
4.1 Experimental Settings
Datasets We utilized several benchmark datasets.
Ofﬁce-31 [22] consists of three domains: Amazon(A), Webcam (W), and DSLR (D) with 31 classes.
Ofﬁce-Home [30] is a more challenging datasetwith four different domains: Artistic (A), Clipart (C), Product (P), and Real-World (R), containing 65classes.
VisDA [21] is a large-scale dataset from synthetic images to real one, with 12 classes.
Interms of the class settings, we follow the experimental protocols by [23].
Baselines We compared UADAL and cUADAL with several baselines by choosing the recent worksin CDA, OSDA, and Universal DA (UniDA).
For CDA, we choose DANN [7] and CDAN [17] toshow that the existing domain adversarial learning for CDA is not suitable for OSDA.
For OSDA, wecompare UADAL to OSBP [23], STA [13], PGL [19], ROS [5], and OSLPP [31].
For UniDA, weconduct the OSDA setting in DANCE [24] and DCC [12].
Implementation We commonly utilize three alternative pre-trained backbones: 1) ResNet-
DenseNet-121 [11], and 3) EfﬁcientNet-B0 [26] to show the robustness on backbone network choices,while using VGGNet [25] on VisDA for fair comparison.
(Implementation details in Appendix 3.1).
Metric We utilize HOS, which is a harmonic mean of OS ∗ , and UNK [5].
OS ∗ is a class-wise averagedaccuracy over known classes, and UNK is an accuracy only for the unknown class.
HOS is suitablefor OSDA because it is higher when performing well in both known and unknown classiﬁcations.
[Table] [Table]
4.2 Experimental Results
Quantitative Analysis Table 1 reports the quantitative performances of UADAL, cUADAL, andbaselines applied to Ofﬁce-31 and Ofﬁce-Home, with three backbone networks.
UADAL andcUADAL show statistically signiﬁcant improvements compared to all existing baselines in both
Ofﬁce-31 and Ofﬁce-Home.
Moreover, there is a large improvement by UADAL from baselines in
EfﬁcientNet-B0 and DenseNet-121, even with fewer parameters.
In terms of the table on ResNet50, for fair comparison, UADAL outperforms the most competitive model, OSLPP (HOS=67.0).
Compared to the domain adversarial learning methods, such as STA and OSBP, UADAL signiﬁcantlyoutperforms in all cases.
UADAL also performs better than other self-supervised learning methodssuch as ROS and DCC.
These performance increments are commonly observed in all three backbonestructures and in two benchmark datasets, which represents the effectiveness of the proposed model.
Table 2 shows the experimental results on VisDA, which is known to have a large domain shiftfrom a synthetic to a real domain.
It demonstrates that UADAL and cUADAL outperform the otherbaselines 1 under the signiﬁcant domain shift.
Moreover, the baselines with the feature alignmentsby the domain adversarial learning, such as STA, OSBP and PGL, outperform the self-supervisedlearning models, ROS, DANCE, and DCC, in most cases.
Therefore, these results support that
OSDA essentially needs the feature alignments, when there are signiﬁcant domain shift.
The detaileddiscussions about the low accuracies of some baselines are in Appendix 3.2.2.
In addition, we observethat cUADAL mostly performs better than UADAL, which shows that cUADAL provides morediscriminative information both on known and unknown by the prediction of the classiﬁer C .
[Equation]
c [Equation] cUADAL
Qualitative Analysis Figure 3 represents the t-SNE visualizations [29] of the learned features by
ResNet-50 (the details in Appendix 3.2.3).
We observe that the target-unknown (red) features fromthe baselines are not segregated from the source (green) and the target-known (blue) features.
Onthe contrary, UADAL aligns the features from the source and the target-known instances accuratelywith clear segregation of the target-unknown features.
In order to investigate the learned featuredistributions quantitatively, Figure 4 provides Proxy A -Distance (PAD) between the feature distributions from STA, ROS, and UADAL.
PAD is an empirical measure of distance between domaindistributions [8] (Details in Appendix 3.2.4), and a higher PAD means clear discrimination betweentwo distributions.
Thus, we calculate PAD between the target-known ( tk ) and the target-unknown( tu ) feature distributions from the feature extractor, G .
We found that PAD from UADAL is muchhigher than STA and ROS.
These results represent that UADAL explicitly segregates the unknownfeatures from the known features in the target domain (the detailed analysis in Appendix 3.2.6).
4.3 Ablation Studies
Effectiveness of Unknown-Aware Feature Alignment We conducted ablation studies to investigate whether the unknown-aware feature alignment is essential for the OSDA problem.
[Figure]
[Figure]
Table 3 represents the experimental results where the feature alignmentprocedure of STA or ROS is replaced by UADAL, maintaining theirweighting schemes, in Ofﬁce-31 with ResNet-50.
In terms of each weighting scheme, UADAL core leads to better performances in both STA and
ROS cases.
Surprisingly, when we replace STA’s known-only matchingby the unknown-aware feature alignment of UADAL, the performanceincreases dramatically, compared to STA itself.
It means that the implicitseparation of STA at the classiﬁcation level is limited to segregate the target-unknown features.
Therefore, explicit segregation by UADAL is an essential part to solve OSDA problem.
Moreover, Figure8 shows the empirical convergence of the proposed sequential optimization as PAD value between p sand p tk and between p s and p tu over training, as we desired, i.e., p s ≈ p tk and p tu ↔ { p tk , p s } .
Superiorities of Open-set Recognition: (Effective Recognition) Figure 6a represents the histogramof the source/target entropy values from E after the early stage of training by Eq.
We alsoprovide the sensitivity analysis on the varying number of iterations for early stage, in Appendix 3.2.5.
Therefore, it demonstrates that our assumption on two modalities of the entropy values holds, robustly.
Furthermore, the subﬁgure in Figure 6a shows the histogram of the ˆ w x by the posterior inference.
Itshows that our open-set recognition , ˆ w x , provides a more clear separation for the target-known (blue)and the target-unknown (red) instances than the entropy values.
Figure 6b represents the histogramof the entropy at the end of the training, which is further split than Figure 6a.
This clear separationimplicitly supports that the proposed feature alignment accomplishes both alignment and segregation .(Informativeness)
Furthermore, we conduct a binary classiﬁcation task by thresholding the weights,ˆ w x , whether it predicts tk or tu correctly.
Figure 7a represents the accuracies over the thresholdvalues when applying the posterior inference or the normalized entropy values itself, in Ofﬁce-31.
It shows that the posterior inference discriminates tk and tu more correctly than using the entropyvalue itself.
Additionally, the posterior inference has similar accuracies over the threshold values(0.2, 0.5, and 0.8), which also indicates a clear separation.
Therefore, we conclude that the posteriorinference provides a more informative weighting procedure than the entropy value itself.(Comparison to Thresholding)
Figure 7b shows that UADAL with the posterior inference outperforms all cases of thresholding the entropy values.
It means that the posterior inference is free to setthresholds.
Moreover, the results demonstrate that UADAL with the posterior inference outperformsthe case of using the normalized entropy value, and its frequent updating (solid line) leads to a betterperformance than only initial updating (dashed line).
It is consistent with the analysis from Figure 6b.
Robust on Openness We conduct the openness experiment to show the robustness on the varyingnumber of the known classes.
Figure 5 represents that UADAL performs better than the baselinesover all cases.
Our open-set recognition does not need to set any thresholds since we utilize the lossinformation of the target domain, holistically.
Therefore, UADAL is robust to the degree of openness.
Conclusion
We propose Unknown-Aware Domain Adversarial Learning (UADAL), which is the ﬁrst approach toexplicitly design the domain adversarial learning for OSDA.
We design a new domain discriminationloss and formulate the sequential optimization for aligning the source and the target-known featureswhile simultaneously segregating the unknown features.
UADAL outperforms all baselines on allbenchmark datasets with various backbone networks, by reporting the state-of-the-art performances.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s
contributions and scope?
[Yes
(b) Did you describe the limitations of your work?
[Yes] Details in Appendix 4.
c) Did you discuss any potential negative societal impacts of your work?
[Yes] Details in
Appendix 4.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes](b) Did you include complete proofs of all theoretical results?
[Yes]3.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[Yes] We provide code
in supplemental material, and URL of data and instructions are provided in Appendix
(b) Did you specify all the training details (e
g., data splits, hyperparameters, how theywere chosen)?
[Yes]
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes]
(d) Did you include the total amount of compute and the type of resources used (e.g., type
(a) If your work uses existing assets, did you cite the creators?
[Yes]
(b) Did you mention the license of the assets?
[Yes]
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes]
See Appendix 3.1.4 and 3.1.5.
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[Yes] We mentioned that we use the public benchmark dataset, in
Section 4.1.
(e) Did you discuss whether the data you are using/curating contains personally identiﬁableinformation or offensive content?
[Yes] We use image object recognition datasets, so
there are no personaly identiﬁable information or offensive content.
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]