C ROSS M ATCH : C ROSS -C LASSIFIER C ONSISTENCY R EGULARIZATION FOR O PEN -S ET S INGLE D OMAIN G ENERALIZATION
A BSTRACT
Single domain generalization (SDG) is a challenging scenario of domain generalization, where only one source domain is available to train the model.
Typical
SDG methods are based on the adversarial data augmentation strategy, which complements the diversity of source domain to learn a robust model.
Existing
SDG methods require the source and target domains to have the same label space.
However, as target domains may contain novel categories unseen in source label space, this assumption is not practical in many real-world applications.
In this paper, we propose a challenging and untouched problem: Open-Set Single Domain
Generalization (OS-SDG), where target domains include unseen categories out of source label space.
The goal of OS-SDG is to learn a model, with only one source domain, to classify a target sample with correct class if it belongs to source label space or assign it to unknown classes.
We design a CrossMatch approach to improve the performance of SDG methods on identifying unknown classes by leveraging a multi-binary classifier.
CrossMatch generates auxiliary samples out of source label space by using an adversarial data augmentation strategy.
We also adopt a consistency regularization on generated auxiliary samples between multibinary classifiers and the model trained by SDG methods, to improve the model’s capability on unknown class identification.
Experimental results on benchmark datasets prove the effectiveness of CrossMatch on enhancing the performance of
SDG methods in the OS-SDG setting.
I NTRODUCTION
Deep neural networks have obtained remarkable success in many classification tasks (Sze et al., 2017), as they can learn discriminative feature representations from data.
These achievements are strongly relied on the independent and identically distributed (i.i.d) assumption (Vapnik, 1992), i.e., the training and testing samples are from the same distribution.
However, in real-world application, the i.i.d assumption is often violated due to the dataset shift problem.
As a result, the performance of deep neural networks degrades notably (Hendrycks & Dietterich, 2019; Recht et al., 2019).
To tackle this challenge, domain adaptation (DA) (Long et al., 2018; Peng et al., 2020; Zhu et al., 2021b;a) is proposed, which aims to mitigate the discrepancy between the source and target domains.
DA assumes that both the source and target domains are accessible during model training stage.
Even though DA has demonstrated promising performance on dealing with the dataset shift problem, it still fails in some practical scenarios when the target domains are unaccessible at the model training stage.
Domain generalization (DG) (Zhao et al., 2020b; Zhou et al., 2021b) is introduced to deal with the dataset shift problem and the absence of target domains.
The goal of DG is to train a model from multiple source domains comprising the same label space, such that the learned model can generalize well to any unseen target domains.
Despite of the success of DG in many real-world applications (Zhou et al., 2021a), it yet fails to a worst-case domain generalization scenario, i.e., single domain generalization (SDG).
In SDG, only one source domain is available to train the model while the learned model need to generalize well on many unseen target domains.
Several recent studies have made significant progress on this challenging yet practical scenario (Volpi et al., 2018;
Zhao et al., 2020a; Qiao et al., 2020;
Fan et al., 2021).
[Figure]
[Table]
One assumption in DG and SDG is that the label spaces of source and target domains are uniform.
However, in some more realistic scenarios such as autonomous driving and biomedical applications, it is possible that the target domains contain novel categories that are unseen in the source label space.
As a result [Table] the learned model cannot tackle target samples from novel categories, which dramatically degrades the robustness of existing DG and SDG methods.
Until recently, this interesting yet tough problem is investigated by Shu et al.
(2021), which proposes a task called open domain generalization (ODG).
ODG assumes that both source and target domains have different label spaces [Table] and multiple source domains are accessible for training.
An ODG model either classifies a target sample with the correct class if it belongs to the multiple source label spaces, or marks it as unknown class.
In this paper, we consider a worst-case open-set problem (Geng et al., 2020) in SDG: only one source domain is accessible for training, and multiple target domains include samples that do not belong to any class in the source label space.
We name this novel and challenging problem as Open-Set Single
Domain Generalization (OS-SDG).
Figure 1 illustrates the relationship between source and target label spaces in DA, DG, SDG, ODG, and OS-SDG.
We also summarize how our problem is different from several related problems in Table 1.
The goal of OS-SDG is to learn a model from a single source domain, which can generalize to unseen target domains with samples from unknown classes.
Inspired by universal domain adaptation (You et al., 2019; Saito et al., 2020; Zhu & Li, 2021), an “unknown” target sample tends to have a larger entropy of the learned model’s output than samples from “known” classes.
We can draw a boundary between “known” and “unknown” classes by using the entropy of the learned model’s outputs.
We evaluate the performance of representative SDG methods (Volpi et al., 2018; Zhao et al., 2020a) in the proposed OS-SDG task.
Results (refer to Figure 5 in Section 4) show that the existing SDG methods have very limited capability on identifying unknown classes in target domains, although they obtain promising results in terms of per-class mean accuracy.
These SDG methods only focus on synthesizing samples from diverse distributions to enrich source domain, and thus they cannot properly identify samples from unknown classes.
The research challenge of modeling and identifying unknown classes in open-set single domain generalization is still untouched.
We propose a novel approach named CrossMatch to address the OS-SDG problem.
First, CrossMatch induces unknown classes information by generating auxiliary samples that are potentially out of the source label space.
These auxiliary samples may not belong to the actual unknown classes in the target domains, as we do not have access to the target label space during training stage.
However, we encourage the auxiliary samples to be far away from the known classes in the source domain, and thus they could still assist in identifying whether a sample belongs to known classes or not.
In particular, we adopt a multi-binary classifier (Liu et al., 2019; Saito & Saenko, 2021) that consists of multiple one-vs-all binary classifiers.
For a given sample, if all the binary classifiers mark it as negative, we believe this sample has a high probability of belonging to an unknown class.
Therefore, this characteristic allows us to design a new loss function to generate auxiliary samples for unknown classes.
Second, with the generated auxiliary samples for unknown classes, we minimize their entropy of multi-binary classifier’s output to help model learn discriminative feature distributions between known and unknown classes, and improve the capability of multi-binary classifier on unknown class identification.
Last, we design a novel consistency regularization to further improve the capability of model on unknown class identification by propagating unknown class information, learned by multi-binary classifier, to the model.
R ELATED W ORK AND B ACKGROUND
Domain Generalization (DG) aims to learn a model from multiple source domains and expects it to generalize over unseen target domains.
Muandet et al.
(2013) proposed a kernel-based method to learn domain invariant feature representations by minimizing the dissimilarity across source domains.
Shankar et al.
(2018) augment source domain by using adversarial gradients obtained from a domain discriminator.
Zhou et al.
(2020) synthesize samples by mapping source samples to pseudo-novel domains with an optimal transport-based distance measure.
Recently, meta-learning has involved in
DG problem (Li et al., 2018; Dou et al., 2019; Du et al., 2020).
The main idea lies in learning the model under a domain shift which is induced by meta-train and meta-test from source domains.
Single Domain Generalization (SDG) is a more challenging and realistic problem than DG.
It assumes the access to a labeled source domain for training, and many unseen target domains for test.
The source and target domains are sampled from different distributions.
The goal of SDG is to learn a model from the source domain, which can generalize well on many unseen target domains.
Adversarial gradient-based augmentation is usually used in SDG for generating new samples to enrich the diversity of source domain.
For instance, Adversarial Data Augmentation (ADA) (Volpi et al., 2018) adopts sign-flipped gradients back-propagated from label classifier to generate new samples.
Maximum-Entropy Adversarial Data Augmentation (MEADA)
(Zhao et al., 2020a) further improves ADA by generating more “hard” samples.
Qiao et al.
(2020) use an auxiliary Wasserstein autoencoder to help ADA generate more “challenging” samples.
Fan et al.
(2021) design an adaptive normalization scheme with ADA to enhance the generalization of learned model.
Open-Set Recognition (OSR) aims to identify unknown classes samples that are completely unseen during the training stage (Geng et al., 2020).
Bendale & Boult (2016) propose a model layer, i.e., OpenMax, to estimate the probability of a test sample belonging to the unknown class.
Padhy et al.
(2020) apply the one-vs-all classifier to identify unknown class samples.
Although many OSR methods have obtained good performance under a strong assumption that the source and target domains have similar distributions, they still fail to address the proposed SS-SDG problem, where the source and target domains are selected from different distributions.
Open-Set Domain Adaptation (OSDA) is proposed by Panareda Busto & Gall (2017) where both source and target domains have private label spaces, respectively, and the common label space is known.
Recently, Saito et al.
(2018) adjust the OSDA setting by claiming no source private label space, which means target label space contains source label space.
Recent OSDA (Liu et al., 2019;
Bucci et al., 2020) methods focus on this challenging setting.
However, OSDA faces two limitations.
First, it relies on an assumption that the target domain is accessible during training stage.
Second,
OSDA methods focus on the scenario that there are only one source domain and one target domain.
Open Domain Generalization (ODA) is proposed by (Shu et al., 2021).
In this problem, a target domain and multiple source domains have different label spaces.
The goal is to learn a model from multiple source domains to correctly classify each target sample into either a known class in the source label space or an unknown class.
They utilize meta-learning framework to learn generalizable representations across domains augmented on both feature-level and label-level.
Our OS-SDG problem is more challenging than the ODA problem, as OS-SDG has only one source domain for training but multiple target domains for test, while ODA has multiple source domains for training and only one target domain for test.
Worst-Case Problem is proposed by (Sinha et al., 2018).
In single domain generalization, we consider to learn the model with source domain by solving the following objective function:
[Equation]
where D represents a distance metric that measures the similarity between source and target domains.
ρ indicates the largest transportation cost of moving source domain D s to target domain D t .
θ denotes model parameters optimized by cross-entropy loss function L ce .
Adversarial Data Augmentation (Volpi et al., 2018) aims to enrich the diversity of source domain by generating a domain D to approximate the unseen target domains D t .
It reformulates worst-case problem (in Eq. 1) into a Lagrangian optimization problem with a fixed penalty parameter γ ≥ 0 :
[Equation]
Here D denotes the Wasserstein metric (Volpi et al., 2018) used to preserve the semantics of the generated samples.
The overall loss function is formulated as:
[Equation]
where L const ( θ g ; D , D s ) =
∥ G ( x ) − G ( x s ) ∥ 22 + ∞ · 1 { y ̸ =
y s }
and G is the feature extractor.
The whole training process is an iterative procedure where two stages are alternated, i.e., maximization stage and minimization stage.
At maximization stage, the new domain D is generated from D s by maximizing L ada with learning rate η :
[Equation]
After the maximization stage, the generated domain D is appended to D s .
At the minimization stage, θ is optimized by minimizing the loss L ce with D s .
Multi-Binary Classifier has proved its capability on unknown class identification in domain adaptation related problems (Liu et al., 2019; Saito & Saenko, 2021).
A multi-binary classier F b with k classes is defined as F b = { F 1 b , . .
, F kb } , where F ib is the i -th one-vs-all binary classifier with output p ib =
F ib ( G ( x ))
∈ R 2 .
We use p ib ( t = 0 | x ) and p ib ( t = 1 | x ) , where p ib ( t = 0 | x ) + p ib ( t = 1 | x ) = 1 , to indicate how likely the sample belongs to the i -th class and other classes, respectively.
Recently,
Saito & Saenko (2021) proposes a hard negative binary classifier sampling strategy to help F b learn a better boundary to identify unknown classes by minimizing the following loss function:
[Equation]
Here the objective is to improve the robustness of multi-binary classifier on unknown class identification by optimizing each binary-classifier with the corresponding hard negative samples.
Furthermore, for unlabeled samples, minimizing their multi-binary entropy loss L bent :
[Equation]
They will either be aligned to source samples or be kept as unknown, and help improve the confidence of model on unknown class identification.
M ETHODOLOGY
In this section, we formally introduce the Open-Set Single Domain Generalization (OS-SDG) task, and then describe the proposed CrossMatch approach in detail.
3. O PEN -S ET S INGLE D OMAIN G ENERALIZATION
Given a labeled source domain D s = { ( x is , y is ) } n s i =1 with label space C s , and multiple unseen target domains D t
= {D 1 t , . . .
, D nt } where D it
= { ( x jt,i , y jt,i ) } n it j =1 with label space C t =
C 1 t = · · · = C
nt .
The source and target domains are sampled from different distributions.
As target domains have novel classes that do not belong to C s , we denote those novel classes as an unknown class space in target
[Figure]
domains, C ut =
C t \C s .
The goal of OS-SDG is to train a model with D s to classify target samples into |C s | + 1 classes, where the novel classes in target domains are treated as one unknown class.
Figure 1 illustrates the relationship of label space between source and target domains in OS-SDG.
The research challenges in OS-SDG are twofold.
First, how can we generate unknown classes samples that could complement sample diversity in the source domain?
Second, what is an effective strategy to improve the model’s capability on unknown class identification?
3. C ROSS M ATCH
The proposed CrossMatch approach generates auxiliary samples for unknown classes out of C s (Sec. 3.2.1) and then exploits those samples to improve the capability of existing SDG methods on unknown class identification (Sec. 3.2.2).
An overview of the proposed CrossMatch approach is shown in Figure 2.
Inspired by the worst-case problem defined in SDG, we extend it to OS-SDG setting based on the
CrossMatch approach, as follows:
[Equation]
where θ = { θ g , θ f } includes the parameters of feature extractor G and multi-class classifier F .
θ f b is the parameters of multi-binary classifier F b .
D =
{D k , D u } denotes the augmented domain consists of known classes domain D k and auxiliary unknown classes domain D u .
L k =
{L ce + L ova } represents those loss functions who use D s to optimize model, i.e., cross-entropy loss
L ce ( θ g , θ f ; x s ) and one-vs-all loss L ova ( θ g , θ f b ; x s ) (in Eq. 5).
L u =
{L bent + α L ccr } includes two loss functions that optimize model with D u , i.e., multi-binary entropy loss L bent ( θ g , θ f b ; x u ) (Eq. 6) and the proposed cross-classifier consistency regularization loss L ccr ( θ g , θ f , θ f b ; x u ) that improve the capability of F on unknown class identification.
L unk ( D s ) (Eq. 13) is a newly proposed loss function for generating auxiliary samples.
D is the Wasserstein metric and ρ is the largest domain discrepancy between two domains.
At minimization stage , the overall loss function is defined as follows:
[Equation]
where α is the hyper-parameter.
At maximization stage , we aim to generate two kinds of samples: known class samples x k ∈ D k and auxiliary samples x u ∈ D u which are far away from the known classes in the source label space.
For known class samples, we consider the following Lagrangian relaxation derived from Eq.
7 with penalty parameter γ :
[Equation]
where D ( D k , D s )
=
L const ( θ g ; D k , D s ) is defined in Eq.
To solve this penalty problem in
Eq.
9, we adopt the same strategy in (Volpi et al., 2018) with the suitable conditions (Boyd et al., 2004) by performing stochastic gradient descent procedures.
We define the robust surrogate loss ϕ ( θ ; x s , y s ) =
sup x k ∈D k
{L ce ( θ ; x k , y s ) − γ L const ( θ g ; x k , x s ) } and have ∇ θ ϕ ( θ ; x s , y s ) = ∇ θ L ce
( θ ; x ∗ k , y s ) , where x ∗ k = arg max x k
∈D k
{L ce ( θ ; x k , y s ) − γ L const ( θ g ; x k , x s ) } is an adversarial perturbation of x s at the current model θ .
We adopt a few iterations to generate known class samples with learning rate η by:
[Equation]
The generation procedure in Eq.
10 is consistent with the single domain generation method
ADA (Volpi et al., 2018), which can be further extended to other single domain generation methods, e.g., MEADA (Zhao et al., 2020a), by adding more loss objectives.
Similar to the way of generating known class samples, we also convert Eq.
7 into a Lagrangian relaxation
to generate auxiliary samples with penalty parameter γ :
[Equation]
By adopting the same strategy as known class sample generation process, we can generate auxiliary samples with the following rule:
[Equation]
where η is the learning rate of stochastic gradient ascent.
The details of L unk and L ccr will be described in Section 3.2.1 and Section 3.2.2, respectively.
Remarks.
CrossMatch aims to tackle the challenges in OS-SDG from two aspects: (i) generation of auxiliary samples for unknown classes, and (ii) improvement of unknown class identification.
The key innovation of the proposed approach lies in jointly encouraging the model to generate auxiliary samples for unknown classes by maximizing the newly designed loss L unk at the maximization stage, and improving the capability of multi-class classifier F on unknown class identification. In particular, we optimize the proposed cross-classifier consistency regularization loss L ccr with generated auxiliary samples for unknown classes at the minimization stage.
3.2. A UXILIARY S AMPLE G ENERATION FOR U NKNOWN C LASSES
The key idea of adversarial data augmentation is to use sign-flipped gradients back-propagated from a label classifier or other related terms (Volpi et al., 2018; Qiao et al., 2020) to generate diversified samples.
Inspired by the structure of multi-binary classifier F b , if a given sample is classified as “other class” by all the binary classifiers in F b , it has a high probability to be associated with unknown classes.
For example, C s contains four known classes (e.g., “phone”, “airplane”, “radio” and “car”) and C ut consists of two unknown classes “truck” and “camera”, as illustrated in Figure 1.
We assume that F b has been well trained in the source domain.
If a target sample belongs to the class “phone”, the corresponding binary classifier should output a high probability to “phone”, while the other binary classifiers will classify it as “other class”.
But, if a target sample belongs to “truck”, all the binary classifiers would classify it as “other class”.
With this insight, we design a novel auxiliary sample generalization loss L unk to generate samples that are possibly out of source label space.
Given a source sample x s with label y s , multi-binary classifier F b and feature extractor G , we denote L unk as:
[Equation]
where p y s b,s ( t = {+
+} | x s ) and p y s b,s ( t = 1 | x s ) represent the probabilities of being y s -th class and other classes from the y s -th binary classifier, respectively.
By maximizing L unk at the maximization stage, it encourages the model to generate new samples that all binary classifiers in F b predict it as “other class”.
Thus, the new generated sample could belong to unknown classes.
As entropy loss function has proved its effectiveness on diversifying the adversarial generated samples (Zhao et al., 2020a), to further enrich the diversity of auxiliary samples, we also incorporate the multi-binary loss L bent and one-vs-all loss L ova to L unk .
3.2. C ROSS -C LASSIFIER C ONSISTENCY R EGULARIZATION
With the generated auxiliary samples for unknown classes, we minimize the multi-binary entropy loss L bent to enhance the capability of multi-binary classifier F b on unknown class identification, and help feature extractor G learn discriminative feature representations, which benefit the multi-class classifier F on unknown class identification.
Each auxiliary sample x u will be aligned to source samples if it belongs to C s , or kept as unknown.
To make full use of unknown classes information learned by F b , we propose the cross-classifier consistency regularization (CCR) loss to propagate those information learned by F b to F .
The key idea lies in encouraging F b and F to produce the similar output distribution, in form of one-vs-all, for each generated auxiliary sample from D u :
[Equation]
where p ib,u ∈ R 2 represents the i -th binary classifier’s output.
p ib ′ ,u indicates the one-vs-all version of multi-class classifier F ’s output p u for i -th class, i.e., p ib ′ ,u = (cid:2) p iu , 1 − p iu (cid:3) , p iu is the probability of x belongs to i -th class.
Different from previous consistency regularization (Laine & Aila, 2017; Sajjadi et al., 2016), which encourages the classifier to produce similar outputs for sample under different augmentations, our CCR encourages different classifiers to generate similar output distributions for a given sample.
3. I NFERENCE
[Figure]
Figure 3 illustrates the inference procedure where the learned model M consists of a feature extractor G and a multi-class classifier F .
The target sample is marked as unknown class if its entropy is larger than the threshold µ .
Otherwise, it will be assigned to a class in the source label space C s .
E XPERIMENTS 4. E XPERIMENTAL S ETUP
We briefly introduce the experimental setup in this section.
More details about the experimental settings and experimental results are provided in the Appendix due to space limit.
Datasets.
(1) Digits comprises of five digits datasets: MNIST (LeCun et al., 1989), SVHN (Netzer et al., 2011), USPS (Hull, 1994), MNIST-M and SYN (Ganin & Lempitsky, 2015).
MNIST is the source domain, and its label space includes numbers from 0 to 4.
The other datasets are treated as target domains, and their target unknown label space C ut contains numbers from 5 to 9.
Office31 (Saenko et al., 2010) contains 31 classes collected from three visually distinct domains:
Amazon, DSLR and Webcam.
The 10 classes shared by Office-31 and Caltech-256 (Gong et al., 2012) form the source label space C s .
In alphabetical order, the last 11 classes are used as target unknown class space C ut .
Due to DSLR and Webcam are relatively small, we only use Amazon as the source domain.
(3) Office-Home (Venkateswara et al., 2017) consists of four domains: Artistic,
Clip Art, Product, and Real-World.
It comprises of 65 categories from four dissimilar domains.
In alphabetic order, we select the first 15 classes as the source label space C s .
The remaining 50 classes are viewed as target unknown label space C ut .
Each domain serves as source domain, and the rest are target domains.
(4) PACS (Li et al., 2017) consists of four domains: Art Paint, Cartoon, Sketch, and Photo.
It has 9,991 images from seven object categories.
Implementation Details.
For Digits , we adopt the ConvNet (LeCun et al., 1989) with architecture conv-pool-conv-pool-fc-fc-softmax , resize all images to 32 × 32, and follow the settings in Volpi et al.
; Zhao et al. (2020a).
For Office31 , Office-Home and PACS , we use an ImageNet-pretrained
ResNet18 (He et al., 2016) as the base network.
Baselines and Metrics.
We adopt the Empirical Risk Minimization (ERM) (Koltchinskii, 2011) as a simple baseline without any consideration of SDG.
We compare with two state-of-the-art
SDG methods, Adversarial Data Augmentation (ADA)
(Volpi et al., 2018) and Maximum-Entropy
Adversarial Data Augmentation (MEADA) (Zhao et al., 2020a).
We also adopt Open Set Domain
[Table]
[Table]
Adaptation by Backpropagation (OSDAP) (Saito et al., 2018) a representative method in open-set domain adaptation and OpenMax (Bendale & Boult, 2016) a typical open-set recognition method as baselines.
Furthermore, we extend ERM, ADA, and MEADA using the proposed CrossMatch (CM) approach (denoted by “+CM”).
We adopt the overall mean accuracy ( acc ), known class accuracy ( acc k ) unknown class accuracy ( acc u ) and h-score ( hs ) (Fu et al., 2020) as evaluation metrics.
4. R ESULTS AND D ISCUSSIONS
Digits and Office31 .
Table 2 shows the results of our approach and baselines on the Digits and
Office31 datasets.
Our proposed method is superior to ADA by more than 19% and 9% with respect to hs on Digits and Office31 , respectively.
Compared with the state-of-the-art method MEADA, which obtains the best acc among all the methods, our method still significantly outperforms it with more than 8% and 16% in terms of hs on Digits and Office31 , respectively.
Meanwhile, our method achieves comparable mean accuracy ( acc ) than baselines.
The strong baseline method OSDAP, uses source and target domains to learn the model, gets better results with regard to acc u in Digits and Office31 datasets.
While our method learns the model without target domain but gets similar performance in terms of hs in Digits , which proves the effectiveness of our method.
Compared with representative open-set recognition method, OpenMax, our method obtains similar results in terms of hs and better performances with regard to acc and acc k in Digits dataset.
OpenMax, without considering the domain shift problem, gets high performance in terms of acc u by wrongly marking most of testing samples as unknown class.
When domain shift is larger in Office31 dataset, we find that OpenMax gets worse performance in terms of acc , acc k , and hs .
Office-Home .
Table 3 shows the acc and hs on Office-Home .
Each time, one domain serves as the source domain and the rest three ones are target domains.
Office-Home is more challenging than the previous two datasets because it contains more target unknown classes than them.
Although the performance of our method on acc is slightly lower than baselines, we observe remarkable improvement of 16 .
22% , and 13 .
13% compared with ERM, ADA, and MEADA in terms of hs , respectively.
Such improvements over baselines demonstrate the effectiveness of our method.
Furthermore, compared with strong baseline OSDAP, our method gets better performance in terms of acc and similar results on hs , demonstrates the efficiency of our method.
OpenMax fails to correctly identify known classes due to significant domain shift existed in Office-Home dataset.
PACS .
We further evaluate the effectiveness of our method on PACS dataset.
As shown in Table 4, our method outperforms all the baselines in terms of acc and hs .
In particular, our method significantly [Table] improves the capability of baselines, i.e., ERM, ADAN and MEADA, on unknown class identification while slightly enhances the averaged accuracy over all classes.
For other two baselines, i.e., OSDAP and OpenMax, we observe the similar phenomenons as Office-Home when large domain gaps are existed among source and target domains.
4. A NALYSIS
L bent + L ova
L unk
L ccr acc acc u hs 52.
✓ 51. 29.36 ✓ ✓ 51. ✓ ✓ 52. 30.55 ✓ ✓ ✓ 51.
On the Effect of L unk and L ccr .
We conduct ablation studies to examine the effectiveness of the proposed L unk (Eq. 13) and
L ccr (Eq. 14) and show the results in Table 5.
The top row denotes the performance of baseline, i.e., MEADA.
Firstly, the middle row demonstrates that simply maximizing the combination of L bent and L ova to generate “auxiliary” samples for unknown class would slightly decrease the performance of baseline. With the proposed L unk , we observe that the performance surpasses the baseline by more than 12% and 6% in terms of acc u and hs , which proves the effectiveness of proposed L unk .
Moreover, we observe that incorporating L ccr can consistently improve the performance of both models in the middle row, which validate the effectiveness of L ccr .
[Figure]
Varying Size of known classes.
To verify the robustness of our method under different sizes of known classes, we conduct experiments on Office-Home where Real-World is selected as the source domain.
The size of known classes varies from 10 to 60.
As shown in Figure 4, our method continuously outperforms
MEADA with significant improvements in terms of hs while obtains similar results as MEADA with respect to acc .
It indicates that our method can consistently improve the capability of SDG methods on unknown class identification.
C ONCLUSION
In this paper, we propose a new domain generalization task, i.e., open-set single domain generalization (OS-SDG), which aims to generalize the model from a single source domain to unseen target domains and simultaneously deal with unknown classes.
We further propose a novel CrossMatch approach to tackle the OS-SDG task, which generates auxiliary samples for unknown classes and improves the capability of unknown class identification with a novel consistency regularization.
Extensive experiments demonstrate that the proposed CrossMatch method largely facilitates the capability of existing single domain generalization methods on unknown class identification. A CKNOWLEDGEMENT
This research is supported by the U.S. Army Research Office Award (W911NF-21-1-0109).
A A PPENDIX
The appendix provides descriptions of algorithm, details for datasets and experimental settings, and additional experimental results.
A. A LGORITHM
Algorithm 1 illustrates details of our proposed method.
Input: Source domain D s = { ( x is , y is ) } n s i =1 and generated auxiliary unknown class domain D u = {} .
Initialized parameters θ g , θ f and θ f b for feature extractor G , multi-class classifier F and multibinary classifier F b .
Output: Learned feature extractor G and multi-class classifier F ; 1: for k = 1 to K do ▷ Run the maximizing procedure K times 2: for t = 1 to T MIN do ▷ Run the minimization stage T MIN times 3:
Sample ( x s , y s ) uniformly from D s 4: ( θ g , θ f ) ← ( θ g , θ f ) − β ∇ ( θ g ,θ f )
L ce ( θ g , θ f ; x s , y s ) 5: ( θ g , θ f b ) ← ( θ g , θ f b ) − β ∇
( θ g ,θ fb )
L ova ( θ g , θ f b ; x s , y s ) 6: if D u is not empty then 7:
Sample x u uniformly from D u 8: ( θ g , θ f b ) ← ( θ g , θ f b ) − β ∇ ( θ g ,θ fb )
L bent ( θ g , θ f b ; x u ) 9: ( θ g , θ f , θ f b ) ← ( θ g , θ f , θ f b ) − β ∇ ( θ g ,θ f ,θ fb ) L ccr ( θ g , θ f , θ f b ; x u ) 10: end if 11: end for 12: for all ( x s , y s ) ∈
D s do 13: x k ← x s and x u ← x s 14: for t = 1 to T MAX do ▷ Run the maximization stage T MAX times 15: x k ← x k + η ∇ x k {L sdg − γ L const ( x k , x s ) } 16: x u ← x u + η ∇ x u (cid:8) L unk + L ova + + L bent + γ L const ( x u , x s ) (cid:9) 17:
Append ( x k , y s ) to D s and x u to D u 18: end for 19: end for 20: end for 21: while not reach maximum steps T do 22:
Sample ( x s , y s ) uniformly from D s 23:
Sample x u uniformly from D u 24: ( θ g , θ f ) ← ( θ g , θ f ) − β ∇ ( θ g ,θ f )
L ce ( θ g , θ f ; x s , y s ) 25: ( θ g , θ f b ) ← ( θ g , θ f b ) − β ∇
( θ g ,θ fb )
L ova ( θ g , θ f b ; x s , y s ) 26: ( θ g , θ f b ) ← ( θ g , θ f b ) − β ∇ ( θ g ,θ fb )
L bent ( θ g , θ f b ; x u ) 27: ( θ g , θ f , θ f b ) ← ( θ g , θ f , θ f b ) − β ∇ ( θ g ,θ f ,θ fb ) L ccr ( θ g , θ f , θ f b ; x u ) 28: end while
A. D ATASETS
Digits consists of 5 different datasets: MNIST (LeCun et al., 1989), SVHN (Netzer et al., 2011),
USPS (Hull, 1994), SYN and MNIST-M (Ganin & Lempitsky, 2015).
Following the setting defined by (Volpi et al., 2018; Zhao et al., 2020a), we select 10,000 images from MNIST as the source domain and the rest datasets are viewed as target domains.
The source label space C s contains numbers from 0 to 4 and target unknow class space C ut includes numbers from 5 to 9.
We resize all the images to 32 × 32 and duplicate their channels to convert all the grayscale images to RGB.
Office31 (Saenko et al., 2010) includes 4,652 images in 31 categories collected from three domains:
Amazon, Webcam and DSLR.
We follow the protocol proposed by (Panareda Busto & Gall, 2017).
We use the 10 classes (back pack, bike, calculator, headphones, keyboard, laptop, monitor, mouse, mug and projector) shared by Office-31 and Caltech-256 (Gong et al., 2012) as the source domain label space C s .
Then in alphabetical order, the last 11 classes (ruler, punchers, stapler, scissors, trash can, tape dispenser, pen, phone, printer, ring binder and speaker) are used as target unknown class space C ut .
Due to the number of images in DSLR and Webcam are relatively small, we only use Amazon as the source domain.
Office-Home (Venkateswara et al., 2017) contains 15,500 images from four domains: Artistic, Clip
Art, Product, and Real-World.
As it comprises of 65 categories from four dissimilar domains, it’s more challenging than Digits and Office-31 .
In alphabetic order, we select the first 15 classes (alarm clock, backpack, battery, bed, bike, bottle, bucket, calculator, calendar, candles, chair, clipboards, computer, couch and curtains) as the source label space C s .
The remaining 50 classes are viewed as target unknown label space C ut .
Each time, one domain is selected as source domain and the rest are viewed as target domain.
PACS (Li et al., 2017) is a recent challenging benchmark for domain generalization that includes four domains: Art Paint, Cartoon, Sketch, and Photo.
It consists of 9,991 images and 7 object categories shared across these domains.
In alphabetic order, we choose the first four classes (dog, elephant, giraffe, and guitar.)
as the source label space C s .
The last three classes (horse, house, and person.) are selected as target unknown label space C ut .
We train the model by one domain and evaluate the learned model by the rest domains
A. E VALUATION M ETRICS .
We adopt the evaluation protocol in Open-Set Domain Adaptation (Panareda Busto & Gall, 2017), where all the target classes out of C s are regarded as one unified unknown class.
We compute the mean accuracy averaged over all classes in |C s | + 1 (i.e., acc ) and the unknown class accuracy (i.e., acc u ) across all target domains.
To better evaluate the capability of methods on unknown class identification, we also adopt the h-score ( hs ) (Fu et al., 2020) metric, hs = 2 ∗ acc s ∗ acc u / ( acc s + acc u ) .
It is the harmonic mean of average per-class accuracy in C s (i.e., acc k ) and acc u .
The h-score value is high only when both acc k and acc u are high.
A. I MPLEMENTATION D ETAILS
We set the same parameters and optimization functions to baselines and ours.
Digits.
Following the setup of (Volpi et al., 2018; Zhao et al., 2020a).
We use ConvNet (LeCun et al., 1989)
( conv-pool-conv-pool-fc-fc-softmax ) as the base model.
The batch size is 32.
We adopt
Adam with learning rate β = 0 .
0001 for minimization stage and SGD with learning rate η = 1 . 0 for maximization stage.
We set T = 10000 , T MIN = 100 , T MAX = 15 , K = 3 , γ = 1 .
and α = 1 .
The reported results are the averaged results based on 10 random experiments.
Office31, Office-Home and PACS.
Referring to the setup of (Zhao et al., 2020a) on PACS , we adopt
ImageNet-pretrained ResNet18 as base network and set the batch size to 32.
We use SGD with learning rate β = 0 .
001 which adjusted by a cosine annealing schedule (Zagoruyko & Komodakis, 2016), the momentum of 0.9 and the weight decay of 0.00005 for minimization stage.
We adopt the
SGD with learning η = 1 . 0 for maximization stage.
We set T = 10000 , T MIN = 100 , T MAX = 15 ,
K = 1 , γ = 1 . 0 and α = 1 .
The reported results are the averaged results based on 3 random experiments.
A. C ONVERGENCE A NALYSIS
We testify the convergence performance of our method under acc and acc u by comparing it against
ADA and MEAD on the Digits and Office31 datasets, respectively.
Figure 5 (a) and (b) illustrate the capability of our method on improving the unknown class identification performance.
We observe that the value of acc u is relatively high at the early training stage.
This phenomenon is induced by the model that is not well-trained and prefers to output high entropy values for testing samples.
Even though the performance of acc u drops during the training stage, our method still outperforms ADA and MEADA in terms of acc u while achieves similar mean accuracy acc . [Figure] [Table]
A. P ARAMETER S ENSITIVITY
We analyze the sensitivity of α in Eq.
8 on Digits datasets by varying it from 0.4 to 1.6.
Table 6 shows performance of our method in terms of acc , acc u , and hs .
Compared ADA and MEADA performance on Digits datasets (in Table 2), our method significantly outperforms them with respect to acc u and hs while obtains comparable results on acc , which proves our method is robust to different choices of α .
We also evaluate the sensitivity of µ in Figure 3 on Digits .
As shown in Table 7, With small value of µ , the inference framework would mark many known classes samples as unknown class.
With large value of µ , the inference framework will reject many unknown class samples to known classes.
A. P ERFORMANCE OF U NKNOWN C LASS I DENTIFICATION AND K NOWN C LASSES C LASSIFICATION
We report the performance of our method and baselines in terms of unknown class accuracy acc u and known classes accuracy acc k on Office-Home and PACS datasets.
More details can be found in
Table 8 and 9.
A. E XPERIMENTAL R ESULTS WITH S TANDARD D EVIATION (STD)
We provide STD value for our method and baselines, i.e., ERM, ADA, and MEADA on four datasets, i.e., Digits , Office31 , Office-Home , and PACS in Table 10, 11, and 12.
[Table]
[Table]
[Table] [Table] [Table] [Table]