C ROSS M ATCH : C ROSS -C LASSIFIER C ONSISTENCY R EGULARIZATION FOR O PEN -S ET S INGLE D OMAIN G ENERALIZATION
Anonymous authors
Paper under double-blind review
A BSTRACT
Single domain generalization (SDG) is a challenging scenario of domain generalization, where only one source domain is available to train the model.
Typical
SDG methods are based on the adversarial data augmentation strategy, which complements the diversity of source domain to learn a robust model.
Existing
SDG methods require the source and target domains to have the same label space.
However, as target domains may contain novel categories unseen in source label space, this assumption is not practical in many real-world applications.
In this paper, we propose a challenging and untouched problem: Open-Set Single Domain
Generalization (OS-SDG), where target domains include unseen categories out of source label space.
The goal of OS-SDG is to learn a model, with only one source domain, to classify a target sample with correct class if it belongs to source label space, or assign it to unknown classes.
We design a CrossMatch approach to improve the performance of SDG methods on identifying unknown classes by leveraging a multi-binary classiﬁer.
CrossMatch generates auxiliary samples out of source label space by using an adversarial data augmentation strategy.
We also adopt a consistency regularization on generated auxiliary samples between multibinary classiﬁers and the model trained by SDG methods, to improve the model’s capability on unknown class identiﬁcation.
Experimental results on benchmark datasets prove the effectiveness of CrossMatch on enhancing the performance of
SDG methods in the OS-SDG setting.
I NTRODUCTION
Deep neural networks have obtained remarkable success in many classiﬁcation tasks (Sze et al., 2017), as they can learn discriminative feature representations from data.
These achievements are strongly relied on the independent and identically distributed (i.i.d) assumption (Vapnik, 1992), i.e., the training and test samples are from the same distribution.
However, in real-world application, the i.i.d assumption is often violated due to the dataset shift problem.
As a result, the performance of deep neural networks degrades notably (Hendrycks & Dietterich, 2019; Recht et al., 2019).
To tackle this challenge, domain adaptation (DA) (Long et al., 2018; Peng et al., 2020) is proposed, which aims to mitigate the discrepancy between the source and target domains.
DA assumes that both the source and target domains are accessible during model training.
Even though DA has demonstrated promising performance on dealing with the dataset shift problem, it still fails in some practical scenarios when the target domains are unaccessible at the model training stage.
Domain generalization (DG) (Zhao et al., 2020b; Zhou et al., 2021b) is introduced to deal with the dataset shift problem and the absence of target domains.
The goal of DG is to train a model from multiple source domains comprising the same label space, such that the learned model can generalize well to any unseen target domains.
Despite of the success of DG in many real-world applications (Zhou et al., 2021a), it yet fails to a worst-case domain generalization scenario, single domain generalization (SDG).
In SDG, only one source domain is available to train the model while the learned model need to generalize well on many unseen target domains.
Several recent studies have made signiﬁcant progress on this challenging yet practical scenario (Volpi et al., 2018;
Zhao et al., 2020a; Qiao et al., 2020;
Fan et al., 2021).
[Figure]
One assumption in DG and SDG is that the label spaces of source and target domains are uniform.
However, in some more realistic scenarios such as autonomous driving and biomedical applications, it is possible that the target domains contain novel categories that are unseen in the source label space.
As a result, the learned model cannot tackle target samples from novel categories, which dramatically degrades the robustness of existing DG and SDG methods.
Until recently, this interesting yet tough problem is investigated by Shu et al.
(2021), which proposes a task called open domain generalization (ODG).
ODG assumes that both source and target domains have different label spaces, and multiple source domains are accessible for training.
An ODG model either classiﬁes a target sample with the correct class if it belongs to the multiple source label spaces, or marks it as unknown class.
In this paper, we consider a worst-case open-set problem (Geng et al., 2020) in SDG: only one source domain is accessible for training, and multiple target domains include samples that do not belong to any class in the source label space.
We name this novel and challenging problem as Open-Set Single
Domain Generalization (OS-SDG).
Figure 1 illustrates the relationship between source and target label spaces in DA, DG, SDG, ODG, and OS-SDG.
We also summarize how our problem is different from several related problems in Table 1.
The goal of OS-SDG is to learn a model from a single source domain, which can generalize to unseen target domains with samples from unknown classes.
Inspired by universal domain adaptation (You et al., 2019; Saito et al., 2020), an “unknown” target sample tends to have a larger entropy of the learned model’s output than samples from “known” classes.
We can draw a boundary between “known” and “unknown” classes by using the entropy of the learned model’s outputs.
We evaluate the performance of representative SDG methods (Volpi et al., 2018; Zhao et al., 2020a) in the proposed OS-SDG task.
Results (refer to Figure 5 in Section 4) show that the existing SDG methods have very limited capability on identifying unknown classes in target domains, although they obtain promising results in terms of per-class mean accuracy.
These
SDG methods only focus on synthesizing samples from diverse distributions to enrich source domain, and thus they cannot properly identify samples from unknown classes.
The research challenge of modeling and identifying unknown classes in open-set single domain generalization is still untouched.
We propose a novel approach named CrossMatch to address the OS-SDG problem.
First, CrossMatch induces unknown class information by generating auxiliary samples that are potentially out of the source label space.
These auxiliary samples may not belong to the actual unknown classes in the target domains, as we do not have access to the target label space during training.
However, we encourage the auxiliary samples to be far away from the known classes in the source domain, and thus they could still assist in identifying whether a sample belongs to known classes or not.
In particular, we adopt a multi-binary classiﬁer (Liu et al., 2019; Saito & Saenko, 2021) that consists of multiple one-vs-all binary classiﬁers.
For a given sample, if all the binary classiﬁers mark it as negative, we believe this sample has a high probability of belonging to an unknown class.
Therefore, this characteristic allows us to design a new loss function to generate auxiliary samples for unknown classes.
Second, with the generated auxiliary samples for unknown classes, we minimize their entropy of multi-binary classiﬁer’s output to help model learn discriminative feature distributions between known and unknown classes, and improve the capability of multi-binary classiﬁer on unknown class identiﬁcation.
Last, we design a novel consistency regularization to improve the capability of model on unknown class identiﬁcation by propagating unknown class information, learned by multi-binary classiﬁer, to the model.
[Table]
R ELATED W ORK AND B ACKGROUND
Domain Generalization (DG) aims to learn a model from multiple source domains and expect it to generalize over unseen target domains.
Muandet et al.
(2013) proposed a kernel-based method to learn domain invariant feature representations by minimizing the dissimilarity across source domains.
Shankar et al.
(2018) augment source domain by using adversarial gradients obtained from a domain discriminator.
Zhou et al
[Table]
(2020) synthesize samples by mapping source samples to pseudo-novel domains with an optimal transport-based distance measure.
Recently, meta-learning has involved in
DG problem (Li et al., 2018; Dou et al., 2019; Du et al., 2020).
The main idea lies in learning the model under a domain shift which is induced by meta-train and meta-test from source domains.
Single Domain Generalization (SDG) is a more challenging and realistic problem than DG.
It assumes the access to a labeled source domain D s =
{ ( x is , y is ) } n s i =1 for training, and many unseen target domains D t
= {D 1 t , . . .
, D nt } for testing, where D it = { ( x jt,i , y jt,i ) } n it j =1 .
The source and target domains are sampled from different distributions.
We denote source and target label spaces as C s and C t , respectively, where C s = C t .
The goal of SDG is to learn a model from D s , which can generalize well on D t .
Adversarial gradient-based augmentation is usually used in SDG for generating new samples to enrich the diversity of source domain.
Firstly, Adversarial Data Augmentation (ADA) (Volpi et al., 2018) adopts sign-ﬂipped gradients back-propagated from label classiﬁer to generate new samples.
Maximum-Entropy Adversarial Data Augmentation (MEADA)
(Zhao et al., 2020a) further improves ADA by generating more “hard” samples.
Qiao et al.
(2020) use an auxiliary
Wasserstein autoencoder to help ADA generate more “challenging” samples.
Fan et al.
(2021) design an adaptive normalization scheme with ADA to enhance the generalization of learned model.
Open-Set Recognition (OSR) aims to identifying unknown classes samples that are completely unseen during training process (Geng et al., 2020).
Bendale & Boult (2016) propose a new model layer, i.e., OpenMax, to estimates the probability of a testing sample belongs to unknown class.
Padhy et al.
(2020) apply one-vs-all classiﬁer to identify unknown class samples.
Although many OSR methods have obtained signiﬁcant achievement under a strong assumption that source and target domains have similar distribution, they still fails to address our proposed SS-SDG problem where source and target domains selected from different distribution.
Open-Set Domain Adaptation (OSDA) is proposed by Panareda Busto & Gall (2017) where both source domain and target domain have private label spaces, respectively, and the common label space is known.
Recently, Saito et al.
(2018) adjust the OSDA setting by claiming no source private label space, which means target label space contains source label space.
Recent OSDA (Liu et al., 2019;
Bucci et al., 2020) methods focus on this challenging setting.
However, OSDA faces two limitations.
First, it relies on an assumption that the target domain is accessible during training phase.
Second,
OSDA methods focus on the scenario that there are only one source domain and one target domain.
Open Domain Generalization (ODA) is proposed by (Shu et al., 2021).
In this problem, a target domain and multiple source domains have different label spaces.
The goal is to learn a model from multiple source domains to correctly classify each target sample into either a known class in the source label space or an unknown class.
They utilize meta-learning framework to learn generalizable representations across domains augmented on both feature-level and label-level.
Our OS-SDG problem is more challenging than the ODA problem and unsuitable for method in (Shu et al., 2021), as OS-SDG has only one source domain for training but multiple target domains for testing while
ODA provides multiple source domains for training and one target domain for testing.
Worst-Case Problem is proposed by (Sinha et al., 2018).
In single domain generalization, we consider to learn the model with source domain by solving the following objective function:
[Equation]
where D represents a distance metric that measures the similarity between source and target domains.
ρ indicates the largest transportation cost of moving D s to D t .
θ denotes model parameters optimized by cross-entropy loss function L ce .
Adversarial Data Augmentation (Volpi et al., 2018) aims to enrich the diversity of source domain by generating a domain D to approximate the unseen target domains D t .
It reformulates worst-case problem (in Eq. 1) into a Lagrangian optimization problem with a ﬁxed penalty parameter γ ≥ 0 :
[Equation]
Here D denotes the Wasserstein metric (Volpi et al., 2018) used to preserve the semantics of the generated samples.
The overall loss function is formulated as:
[Equation]
where L const ( θ g ; D , D s ) =
(cid:107) G ( x ) − G ( x s ) (cid:107) 22 + ∞ · 1 { y (cid:54) =
y s } .
The whole training process is an iterative procedure where two stages are alternated, i.e., maximization stage and minimization stage.
At maximization stage, the new domain D is generated from D s by maximizing L ada :
[Equation]
[Equation]
After the maximization stage, the generated domain D is appended to D s .
At the minimization stage, θ is optimized by minimizing the loss L ce with D s .
Multi-Binary Classiﬁer has proved its capability on unknown class identiﬁcation in domain adaptation related problems (Liu et al., 2019; Saito & Saenko, 2021).
A multi-binary classier F b with k classes is deﬁned as F b = { F 1 b , . .
, F kb } , where F ib is the i -th one-vs-all binary classiﬁer with output p ib =
F ib ( G ( x ))
∈ R 2 .
We use p ib ( t = 0 | x ) and p ib ( t = 1 | x ) , where p ib ( t = 0 | x ) + p ib ( t = 1 | x ) = 1 , to indicate how likely the sample belongs to the i -th class and other classes, respectively.
Recently,
Saito & Saenko (2021) proposes a hard negative binary classiﬁer sampling strategy for multi-binary classiﬁer, which helps F b learn a better boundary to identify unknown classes by minimizing the following loss function:
[Equation]
Here the objective is to improve the robustness of multi-binary classiﬁer on unknown class identiﬁcation by optimizing each binary-classiﬁer with the corresponding hard negative samples.
Furthermore, for unlabeled samples, minimizing their multi-binary entropy loss L bent :
[Equation]
They will either be aligned to source samples or be kept as unknown, and help improve the conﬁdence of model on unknown class identiﬁcation.
M ETHODOLOGY
In this section, we formally introduce the Open-Set Single Domain Generalization (OS-SDG) task, and then describe the proposed CrossMatch approach in detail.
3. O PEN -S ET S INGLE D OMAIN G ENERALIZATION
Given a labeled source domain D s = { ( x is , y is ) } n s i =1 with label space C s , and multiple unseen target domains D t
= {D 1 t , . . .
, D nt } where D it
= { ( x jt,i , y jt,i ) } n it j =1 with label space C t =
C 1 t = · · · = C
nt .
The source and target domains are sampled from different distributions.
As target domains have novel classes that do not belong to C s , we denote those novel classes as an unknown class space in target
[Figure]
G x u …… ..
Multi-Binary Classiﬁer ce L
Multi-Class Classiﬁer |C | s
F 1b
F b |C | S
F g u ccr L ova L ent L b 0.6 0.
G …… ..
Multi-Binary Classiﬁer
Multi-Class Classiﬁer |C | s
F 1b
F unk L 0.6 0.
sdg L
F b |C | S x s g s g s p s p
bs g s g s p s p bs x s g u p bu p u x k x u domains, C ut
=
C t \C s .
The goal of OS-SDG is to train a model with D s to classify target samples into |C s | + 1 classes, where the novel classes in target domains are treated as one unknown class.
Figure 1 illustrates the relationship of label space between source and target domains in OS-SDG.
The research challenges in OS-SDG are twofold.
First, how can we generate unknown classes samples that could complement sample diversity in the source domain?
Second, what is an effective strategy to improve the model’s capability on unknown class identiﬁcation?
3. C ROSS M ATCH
The proposed CrossMatch approach generates auxiliary samples for unknown classes out of C s (Sec. 3.2.1) and then exploits those samples to improve the capability of existing SDG methods on unknown class identiﬁcation (Sec. 3.2.2).
An overview of the proposed CrossMatch approach is shown in Figure 2.
Inspired by the worst-case problem deﬁned in SDG, we extend it to OS-SDG setting based on the
CrossMatch approach, as follows:
[Equation]
where θ = { θ g , θ f } includes the parameters of feature extractor G and multi-class classiﬁer F .
θ f b is the parameters of multi-binary classiﬁer F b .
D =
{D k , D u } denotes the augmented domain consists of known classes domain D k and auxiliary unknown classes domain D u .
L k =
{L ce + L ova } represents those loss functions who use D s to optimize model, i.e., cross-entropy loss
L ce ( θ g , θ f ; x s ) and one-vs-all loss L ova ( θ g , θ f b ; x s ) (in Eq. 5).
L u =
{L bent + α L ccr } includes a loss function that optimize model with D u , i.e., multi-binary entropy loss L bent ( θ g , θ f b ; x u ) (Eq. 6) and the proposed cross-classiﬁer consistency regularization loss L ccr ( θ g , θ f , θ f b ; x u ) that improves the capability of F on unknown class identiﬁcation.
L unk ( D s ) (Eq. 13) is a newly proposed loss function for generating auxiliary samples.
D is the Wasserstein metric and ρ is the largest domain discrepancy between two domains.
At minimization stage , the overall loss function is deﬁned as follows:
[Equation]
where α is the hyper-parameter.
At maximization stage , we aim to generate two kinds of samples: known classes samples x k ∈ D k and auxiliary samples x u ∈ D u which are far away from the known classes in the source domain.
For known class samples, we consider the following Lagrangian relaxation derived from Eq.
7 with penalty parameter γ :
[Equation]
where D ( D k , D s )
=
L const ( θ g ; D k , D s ) is deﬁned in Eq.
To solve this penalty problem in
Eq.
9, we adopt the same strategy in (Volpi et al., 2018) with the suitable conditions (Boyd et al., 2004) by performing stochastic gradient descent procedures.
We deﬁne the robust surrogate loss φ ( θ ; x s , y s ) =
sup x k ∈D k
{L ce ( θ ; x k , y s ) − γ L const ( θ g ; x k , x s ) } and have ∇ θ φ ( θ ; x s , y s ) =
∇ θ L ce
( θ ; x ∗ k , y s ) , where x ∗ k = arg max x k
∈D k
{L ce ( θ ; x k , y s ) − γ L const ( θ g ; x k , x s ) } is an adversarial perturbation of x s at the current model θ .
We adopt a few iterations to generate known class samples by:
[Equation]
The generation procedure in Eq.
10 is consistent with the single domain generation method
ADA (Volpi et al., 2018), which can be further extended to other single domain generation methods, e.g., MEADA (Zhao et al., 2020a), by modifying the loss L ce .
Similar to the way of generating known class samples, we also convert Eq.
7 into a Lagrangian relaxation
to generate auxiliary samples with penalty parameter γ :
[Equation]
By adopting the same strategy as known class sample generation process, we can generate auxiliary samples with the following rule:
[Equation]
where η is the learning rate of stochastic gradient ascent.
The details of L unk and L ccr will be described in Section 3.2.1 and Section 3.2.2, respectively.
Remarks.
CrossMatch aims to tackle the challenges in OS-SDG from two aspects: generation of auxiliary samples for unknown classes and improvement of unknown class identiﬁcation.
The key innovation of the proposed approach lies in jointly encouraging model to generate auxiliary samples for unknown classes by maximizing our new designed loss L unk at maximization stage, and improving the capability of multi-class classiﬁer F on unknown class identiﬁcation by optimizing our proposed cross-classiﬁer consistency regularization loss L ccr with generated auxiliary samples for unknown classes at the minimization stage.
3.2. A UXILIARY S AMPLE G ENERATION FOR U NKNOWN C LASSES
The key idea of adversarial data augmentation is to use sign-ﬂipped gradients back-propagated from a label classiﬁer or other related terms (Volpi et al., 2018; Qiao et al., 2020) to generate diversiﬁed samples.
Inspired by the structure of multi-binary classiﬁer F b , if a given sample is classiﬁed as ‘other class’ by all the binary classiﬁers in F b , it has a high probability to be associated with unknown classes.
For example, C s contains four known classes (e.g., ‘phone’, ‘airplane’, ‘radio’ and ‘car’) and
C u t consist of two unknown classes ‘truck’ and ‘camera’, as illustrated in Figure 1.
We assume that
F b has been well trained in the source domain.
If a target sample belongs to the class ’phone’, the corresponding binary classiﬁer should output a high probability to ‘phone’, while the other binary classiﬁers will classify it as ‘other class’.
But, if a target sample belongs to ‘truck’, all the binary classiﬁers would classify it as ’other class’.
With this insight, we design a novel auxiliary sample generalization loss L unk to generate samples that are possibly out of source label space.
Given a source sample x s with label y s , multi-binary classiﬁer F b and feature extractor G , we denote L unk as:
[Equation]
where p yb ( t = 0 | x ) and p yb ( t = 1 | x ) represent the probabilities of being y -th class and other classes from the y -th binary classiﬁer, respectively.
By maximizing L unk at the maximization stage, it encourages the model to generate new samples that all binary classiﬁers in F b predict it as other class.
Thus, the new generated sample could belong to unknown classes.
As entropy loss function has proved its effectiveness on diversifying the adversarial generated samples (Zhao et al., 2020a), we also add multi-binary loss
L bent to L unk .
3.2. C ROSS -C LASSIFIER C ONSISTENCY R EGULARIZATION
With the generated auxiliary samples for unknown classes, we minimize the multi-binary entropy loss L bent to enhance the capability of multi-binary classiﬁer F b on unknown class identiﬁcation, and help feature extractor G learn discriminative feature representations to beneﬁt multi-class classiﬁer F on unknown class identiﬁcation.
Each auxiliary sample x u will be aligned to source samples if it really belongs to C s , or kept as unknown.
To make full use of unknown classes information learned by F b , we propose the cross-classiﬁer consistency regularization loss, termed as cross-classiﬁer consistency regularization (CCR), to propagate those information learned by F b to F .
The key idea lies in encouraging F b and F to produce the similar output distribution, in form of one-vs-all, for each generated auxiliary sample from D u :
[Equation]
where p ib ∈ R 2 represents the i -th binary classiﬁer’s output.
p ib (cid:48) indicates the one-vs-all version of multi-class classiﬁer F ’s output p , i.e., p ib (cid:48)
= (cid:2) p
i , 1 − p
i (cid:3) , p i is the probability of x belongs to i -th class.
Different from previous consistency regularization (Laine & Aila, 2017; Sajjadi et al., 2016), which encourages classiﬁer to produce similar outputs for sample under different augmentations,
our
CCR encourages different classiﬁers to generate similar output distributions for given sample.
3. I NFERENCE
[Figure]
Figure 3 illustrates the inference procedure where the learned model M consists of a feature extractor G and a multi-class classiﬁer F .
The target sample is marked as unknown class if its entropy is larger than the threshold µ .
Otherwise, it will be assigned to a class in the source label space C s .
E XPERIMENTS 4. E XPERIMENTAL S ETUP
We brieﬂy introduce the experimental setup in this section.
More details about the experimental settings and results on additional datasets are provided in the Appendix due to space limit.
Datasets.
(1) Digits comprises of ﬁve digits datasets: MNIST (LeCun et al., 1989), SVHN (Netzer et al., 2011), USPS (Hull, 1994), MNIST-M and SYN (Ganin & Lempitsky, 2015).
MNIST is the source domain, and its label space includes numbers from 0 to 4.
The other datasets are treated as target domains, and their target unknown label space C ut contains numbers from 5 to 9.
Ofﬁce31 (Saenko et al., 2010) contains 31 classes collected from three visually distinct domains:
A , D and W .
The 10 classes shared by Ofﬁce-31 and Caltech-256 (Gong et al., 2012) form the source label space C s .
In alphabetical order, the last 11 classes are used as target unknown class space C ut .
Due to D and W are relatively small, we only use A as the source domain.
(3) OfﬁceHome (Venkateswara et al., 2017) consists of four domains: Artistic , Clip Art , Product , and
Real-World .
It comprises of 65 categories from four dissimilar domains.
In alphabetic order, we select the ﬁrst 15 classes as the source label space C s .
The remaining 50 classes are viewed as target unknown label space C ut .
Each domain serves as source domain, and the rest are target domains.
PACS (Li et al., 2017) consists of four domains: Art Paint, Cartoon, Sketch, and Photo.
It has 9, images from seven object categories.
Implementation Details.
For Digits , we adopt the ConvNet (LeCun et al., 1989) with architecture conv-pool-conv-pool-fc-fc-softmax , resize
all
images
to
32 × 32, and follow the settings
in Volpi
et al.
Zhao
et al.
(2020a
For Ofﬁce
Ofﬁce
-Home
and
PACS , we use an ImageNet pretrained
ResNet
(He et al
as the base network.
Baselines and Metrics.
We adopt the Empirical Risk Minimization (ERM) (Koltchinskii, 2011) as a simple baseline without any consideration of SDG.
We compare with two state-of-the-art
SDG methods, Adversarial Data Augmentation (ADA)
(Volpi et al., 2018) and Maximum-Entropy
Adversarial Data Augmentation (MEADA) (Zhao et al., 2020a).
We also adopt Open Set Domain
[Table]
[Table]
Adaptation by Backpropagation (OSDAP) (Saito et al., 2018) a representative method in open-set domain adaptation and OpenMax (Bendale & Boult, 2016) a typical open-set recognition method as baselines.
Furthermore, we extend ERM, ADA, and MEADA using the proposed CrossMatch (CM) approach (denoted by “+CM”).
We adopt the overall accuracy ( acc ), unknown class accuracy ( acc u ) and h-score ( hs ) (Fu et al., 2020) as evaluation metrics.
4. R ESULTS AND D ISCUSSIONS
Digits and Ofﬁce31 .
Table 2 shows the results of our approach and baselines on the Digits and
Ofﬁce31 datasets in terms of mean accuracy ( acc ), unknown class accuracy ( acc u ) and h-score ( hs ).
ERM is a simple baseline without any consideration of adversarial data augmentation.
Our proposed method is superior to ADA by more than 19% and 9% with respect to hs on Digits and Ofﬁce31 , respectively.
Compared with the state-of-the-art method MEADA, which obtains the best acc among all the methods, our method still signiﬁcantly outperforms it with more than 8% and 16% in terms of hs on Digits and Ofﬁce31 , respectively.
Meanwhile, our method achieves comparable mean accuracy ( acc ) than baselines.
The strong baseline method OSDAP, uses source and target domains to learn the model, gets better results with regard to acc u in Digits and Ofﬁce31 datasets.
While our method learns the model without target domain but gets similar performance in terms of hs in Digits , which proves the effectiveness of our method.
Compared with representative open-set recognition method,
OpenMax, our method obtains similar results in terms of hs and better performances with regard to acc and acc s in Digits dataset.
Due to domain shift problem, OpenMax gets high performance in terms of acc u by marking most of testing samples as unknown class.
When domain shift is larger in
Ofﬁce31 dataset, we ﬁnd that OpenMax get worse performance in terms of acc , acc s , and hs .
Ofﬁce-Home .
Table 3 shows the acc and hs on Ofﬁce-Home .
Each time, one domain serves as the source domain and the rest three ones are target domains.
Ofﬁce-Home is more challenging than the previous two datasets because it contains more target unknown classes than them.
Although the performance of our method on acc is slightly lower than baselines, we observe remarkable improvement of 16 .
22% , and 13 .
13% compared with ERM, ADA, and MEADA in terms of hs , respectively.
Such improvements over baselines demonstrate the effectiveness of our method.
Furthermore, compared with strong baseline OSDAP, our method gets better performance in terms of acc and similar results on hs , demonstrates the efﬁciency of our method.
OpenMax fails to correctly identify known classes due to signiﬁcant domain shift existed in Ofﬁce-Home dataset.
[Table]
PACS .
We further evaluate the effectiveness of our method on PACS dataset.
As shown in Table 4, our method outperforms all the baselines in terms of acc and hs .
In particular, our method signiﬁcantly improve the capability of baselines, i.e., ERM, ADAN and MEADA, on unknown class identiﬁcation while slightly enhance the averaged accuracy over all classes.
For other two baselines, i.e., OSDAP and OpenMax, we observe the similar phenomenon as Ofﬁce-Home where large domain gaps are existed among source and target domains.
4. A NALYSIS
[Table]
On the Effect of L unk and L ccr .
We investigate the efﬁcacy of the proposed L unk (Eq. 13) and
L ccr (Eq. 14) on Digits datasets with MEADA method.
To prove the advantage of L unk on generating auxiliary samples for unknown classes, we compare it with a baseline L (cid:48) unk
=
L ova + L bent .
The major difference between L unk (cid:48) and L unk is that the former aims to generate samples that are hard for F b to separate, while the latter prefers to generate those out of C s .
As shown in Table 5, the ﬁrst row denotes the performance of MEADA.
Firstly, comparing results in the ﬁrst three rows, we ﬁnd that our proposed L unk outperforms L (cid:48) unk in terms of acc u and hs , demonstrating the advantage of L unk on generating auxiliary samples for unknown classes.
Adding L (cid:48) unk to MEADA degrades the performance.
Secondly, the performance of both L unk and L (cid:48) unk are improved by adding L ccr , which validates the effectiveness of L ccr .
[Figure]
Varying Size of known classes.
To verify the robustness of our method under different sizes of known classes, we conduct experiments on Ofﬁce-Home where Real-World is selected as the source domain.
The size of known classes varies from 10 to 60.
As shown in Figure 4, our method continuously outperforms MEADA with signiﬁcant improvements under different sizes of known classes.
In addition, our method obtains similar results as MEADA in terms of acc .
It indicates that our method can improve the capability of single domain generalization methods on unknown class identiﬁcation.
C ONCLUSION
In this paper, we propose a new domain generalization task, i.e., open-set single domain generalization (OS-SDG), which aims to generalize models from a single source domain to unseen target domains and simultaneously deals with unknown classes.
We further propose a novel CrossMatch approach to tackle the OS-SDG task, which generates auxiliary samples for unknown classes and improves the capability of unknown classes identiﬁcation with a novel consistency regularization.
Extensive experiments demonstrate that the proposed CrossMatch method largely facilitates the capability of existing single domain generalization methods on unknown class identiﬁcation.
A A PPENDIX
The appendix provides descriptions of algorithm, details for datasets and experimental settings, and additional experimental results.
A. A LGORITHM
Algorithm 1 illustrates details of our proposed method.
Input: Source domain D s = { ( x is , y is ) } n s i =1 and generated auxiliary unknown class domain D u = {} .
Initialized parameters θ g , θ f and θ f b for feature extractor G , multi-class classiﬁer F and multibinary classiﬁer F b .
Output: Learned feature extractor G and multi-class classiﬁer F ; 1: for k = 1 to K do (cid:46) Run the maximizing procedure K times 2: for t = 1 to T MIN do (cid:46)
Run the minimization stage T MIN times 3:
Sample ( x s , y s ) uniformly from D s 4: ( θ g , θ f ) ← ( θ g , θ f ) − β ∇ ( θ g ,θ f )
L ce ( θ g , θ f ; x s , y s ) 5: ( θ g , θ f b ) ← ( θ g , θ f b ) − β ∇
( θ g ,θ fb )
L ova ( θ g , θ f b ; x s , y s ) 6: if D k is not empty then 7:
Sample x u uniformly from D u 8: ( θ g , θ f b ) ← ( θ g , θ f b ) − β ∇ ( θ g ,θ fb )
L bent ( θ g , θ f b ; x u ) 9: ( θ g , θ f , θ f b ) ← ( θ g , θ f , θ f b ) − β ∇ ( θ g ,θ f ,θ fb ) L ccr ( θ g , θ f , θ f b ; x u ) 10: end if 11: end for 12: for all ( x s , y s ) ∈
D s do 13: x k ← x s and x u ← x s 14: for t = 1 to T MAX do (cid:46)
Run the maximization stage T MAX times 15: x k ← x k + η ∇ x k {L sdg − γ L const ( x k , x s ) } 16: x u ← x u + η ∇ x u {L unk + γ L const ( x u , x s ) } 17:
Append ( x k , y s ) to D s and x u to D u 18: end for 19: end for 20: end for 21: while not reach maximum steps T do 22:
Sample ( x s , y s ) uniformly from D s 23:
Sample x u uniformly from D u 24: ( θ g , θ f ) ← ( θ g , θ f ) − β ∇ ( θ g ,θ f )
L ce ( θ g , θ f ; x s , y s ) 25: ( θ g , θ f b ) ← ( θ g , θ f b ) − β ∇
( θ g ,θ fb )
L ova ( θ g , θ f b ; x s , y s ) 26: ( θ g , θ f b ) ← ( θ g , θ f b ) − β ∇ ( θ g ,θ fb )
L bent ( θ g , θ f b ; x u ) 27: ( θ g , θ f , θ f b ) ← ( θ g , θ f , θ f b ) − β ∇ ( θ g ,θ f ,θ fb ) L ccr ( θ g , θ f , θ f b ; x u ) 28: end while
A. D ATASETS
Digits consists of 5 different datasets: MNIST (LeCun et al., 1989), SVHN (Netzer et al., 2011),
USPS (Hull, 1994), SYN and MNIST-M (Ganin & Lempitsky, 2015).
Following the setting deﬁned by (Volpi et al., 2018; Zhao et al., 2020a), we select 10,000 images from MNIST as the source domain and the rest datasets are viewed as target domains.
The source label space C s contains numbers from zero to four and target unknow class space C ut includes numbers from ﬁve to nine.
We resize all the images to 32 × 32 and duplicate their channels to convert all the grayscale images to RGB.
Ofﬁce31 (Saenko et al., 2010) includes 4,652 images in 31 categories collected from three domains:
Amazon (A), Webcam (W) and DSLR (D).
We follow the protocol proposed by (Panareda Busto & Gall, 2017).
We use the 10 classes (back pack, bike, calculator, headphones, keyboard, laptop,
[Figure]
[Table]
monitor, mouse, mug and projector) shared by Ofﬁce-31 and Caltech-256 (Gong et al., 2012) as the source domain label space C s .
Then in alphabetical order, the last 11 classes (ruler, punchers, stapler, scissors, trash can, tape dispenser, pen, phone, printer, ring binder and speaker) are used as target unknown class space C ut .
Due to the number of images in DSLR and Webcam are relatively small, we only use Amazon as the source domain.
Ofﬁce-Home (Venkateswara et al., 2017) contains 15,500 images from four domains: Artistic (Ar),
Clip Art (Cl), Product (Pr), and Real-World (Rw).
As it comprises of 65 categories from four dissimilar domains, it’s more challenging than Digits and Ofﬁce-31 .
In alphabetic order, we select the ﬁrst 15 classes (alarm clock, backpack, battery, bed, bike, bottle, bucket, calculator, calendar, candles, chair, clipboards, computer, couch and curtains) as the source label space C s .
The remaining 50 classes are viewed as target unknown label space C ut .
Each time, one domain is selected as source domain and the rest are viewed as target domain.
PACS (Li et al., 2017) is a recent challenging benchmark for domain generalization that includes four domains: Art Paint, Cartoon, Sketch, and Photo.
It consists of 9,991 images and seven object categories shared across these domains.
In alphabetic order, we choose the ﬁrst four classes (dog, elephant, giraffe, and guitar.)
as the source label space C s .
The last three classes (horse, house, and person.) are selected as target unknown label space C ut .
We train the model by one domain and evaluate the learned model by the rest domains
A. E VALUATION M ETRICS .
We adopt the evaluation protocol in Open-Set Domain Adaptation (Panareda Busto & Gall, 2017), where all the target classes out of C s are regarded as one uniﬁed unknown class.
We compute the mean accuracy averaged over all classes in |C s | + 1 (i.e., acc ) and the unknown class accuracy (i.e., acc u ) across all target domains.
To better evaluate the capability of methods on unknown class identiﬁcation, we also adopt the h-score ( hs ) (Fu et al., 2020) metric, hs = 2 ∗ acc s ∗ acc u / ( acc s + acc u ) .
It is the harmonic mean of average per-class accuracy in C s (i.e., acc
[Table] [Table]
A. I MPLEMENTATION D ETAILS
We set the same parameters and optimization functions to baselines and ours.
Digits.
Following the setup of (Volpi et al., 2018; Zhao et al., 2020a).
We use ConvNet (LeCun et al., 1989)
( conv-pool-conv-pool-fc-fc-softmax ) as the base model.
The batch size is 32.
We adopt
Adam with learning rate β = 0 .
0001 for minimization stage and SGD with learning rate η = 1 . 0 for maximization stage.
We set T = 10000 , T MIN = 100 , T MAX = 15 , K = 3 , γ = 1 .
and α = 1 .
The reported results are the averaged results based on 10 random experiments.
Ofﬁce31, Ofﬁce-Home and PACS.
Referring to the setup of (Zhao et al., 2020a) on PACS , we adopt
ImageNet-pretrained ResNet18 as base network and set the batch size to 32.
We use SGD with learning rate β = 0 .
001 which adjusted by a cosine annealing schedule (Zagoruyko & Komodakis, 2016), the momentum of 0.9 and the weight decay of 0.00005 for minimization stage.
We adopt the
SGD with learning η = 1 . 0 for maximization stage.
We set T = 10000 , T MIN = 100 , T MAX = 15 ,
K = 1 , γ = 1 . 0 and α = 1 .
The reported results are the averaged results based on 3 random experiments.
A. C ONVERGENCE A NALYSIS
We testify the convergence performance of our method under acc and acc u by comparing it against
ADA and MEAD on the Digits and Ofﬁce31 datasets.
Figure 5(b) and (c) illustrate the capability of our method on improving the unknown class identiﬁcation performance.
We observe that the value of acc u is relatively high at the early training stage.
This phenomenon is induced by the model that is not well-trained and prefers to output high entropy values for testing samples.
Even though the performance of acc u drops during the training process, our method still outperforms ADA and
MEADA in terms of acc u while achieving similar mean accuracy acc .
A. P ARAMETER S ENSITIVITY
We analyze the sensitivity of α in Eq.
8 on Digits datasets by varying it from 0.4 to 1.6.
Table 6 shows performance of our method in terms of acc , acc u , and hs .
Compared ADA and MEADA performance on Digits datasets (in Table 2), our method signiﬁcantly outperforms them with respect to acc u and hs while obtains comparable results on acc , which proves our method is robust to different choices of
[Table] [Table]
α .
We also evaluate the sensitivity of µ in Figure 3 on Digits .
As shown in Table
[Table]
With small value of µ , the inference framework would mark many known classes samples as unknown class.
With large value of µ , the inference framework will reject many unknown class samples to known classes.
A. P ERFORMANCE OF U NKNOWN C LASS I DENTIFICATION AND K NOWN C LASSES C LASSIFICATION
We report the performance of our method and baselines in terms of unknown class accuracy acc u and known classes accuracy acc s on Ofﬁce-Home and PACS datasets.
More details can be found in
Table 8 and 9.
A. E XPERIMENTAL R ESULTS WITH S TANDARD D EVIATION (STD)
We provide STD value for our method and baselines, i.e., ERM, ADA, and MEADA on four datasets, i.e., Digits , Ofﬁce31 , Ofﬁce-Home , and PACS in Table 10, 11, and 12.
[Table] [Table]