C OLLABORATIVE T RAINING OF B ALANCED R ANDOM F ORESTS FOR O PEN S ET D OMAIN A DAPTATION
Anonymous authors
Paper under double-blind review
A BSTRACT
In this paper, we introduce a collaborative training algorithm of balanced random forests for domain adaptation tasks which can avoid the overﬁtting problem.
In real scenarios, most domain adaptation algorithms face the challenges from noisy, insufﬁcient training data.
Moreover in open set categorization, unknown or misaligned source and target categories adds difﬁculty.
In such cases, conventional methods suffer from overﬁtting and fail to successfully transfer the knowledge of the source to the target domain.
To address these issues, the following two techniques are proposed.
First, we introduce the optimized decision tree construction method, in which the data at each node are split into equal sizes while maximizing the information gain.
Compared to the conventional random forests, it generates larger and more balanced decision trees due to the even-split constraint, which contributes to enhanced discrimination power and reduced overﬁtting.
Second, to tackle the domain misalignment problem, we propose the domain alignment loss which penalizes uneven splits of the source and target domain data.
By collaboratively optimizing the information gain of the labeled source data as well as the entropy of unlabeled target data distributions, the proposed CoBRF algorithm achieves signiﬁcantly better performance than the state-of-the-art methods.
The proposed algorithm is extensively evaluated in various experimental setups in challenging domain adaptation tasks with noisy and small training data as well as open set domain adaptation problems, for two backbone networks of AlexNet and ResNet-50.
I NTRODUCTION
In recent years, domain adaptation has been researched as it can help to solve major difﬁculties in the real world.
Due to the huge overhead in labeling large-scale training data, it is desirable if an existing network can be adapted to different target domains.
More importantly, it is common that the training dataset for adaptation is noisy and small, or the labels in the target domain do not match with the source or even unknown.
These are inherent challenges in the domain adaptation problem as in real world it is common for the data to contain such class bias, noise and unlabeled data.
However, in practice, since the adapted networks are often overﬁtted to the provided source data or the data distribution of the target domain is frequently quite different from the source, they do not perform well to the target domain.
To properly deal with these real-world conditions with insufﬁcient information, it is critical to learn the shared data distribution that is effective both in the source and target domain.
To this end, we propose the collaborative training algorithm of balanced random forest (CoBRF) to mitigate the challenging problems such as noisy labels, lack of training data, and misaligned or unknown categories (open set categorization).
In random forests, multiple decision trees are learned by optimizing the information gain for the randomly selected subset features at each node split.
Since random forests ensemble the internal decision trees, they are more robust to noise and overﬁtting problem than single decision trees.
To improve the robustness of the random forests, we take one step further by balancing the decision trees, i.e., maximizing the number of leaf nodes for the same tree depth.
Our method builds more balanced decision trees by enforcing the sizes of the data in the left and right child nodes to be equal.
While this split strategy is not locally optimal in terms of information gain, the resulting decision trees have far more leaf nodes, and it endows more expressive power which can be helpful in dealing with noise and unseen data or classes.
It also helps to avoid overﬁtting as it prevents a node committing too early for a speciﬁc pattern, or in other words, it postpones the decision as late as possible so that various discriminant information in the training data can be fully considered.
To enforce even splits while maintaining the discriminability, the CoBRF uses the hyperplanes estimated by the linear support vector machine (SVM).
First, it randomly assigns the classes in the nodes to binary pseudo labels and equalizes the sizes of two pseudo classes by randomly removing data in the larger class.
Then a linear classiﬁer is found by SVM, and its hyperplane is translated until the data sizes on both sides are equal.
In a sense, it ﬁnds the even split of the data projected onto the normal direction of the hyperplane and places the hyperplane there.
The node split by the translated hyperplane is simple yet effective.
The ablation study in Sec.
4.2 conﬁrms that the CoBRF boosts the performance compared to the baseline random forests.
Since the above training process only considers maximizing the information gain of labeled data in the source domain, which is referred to ‘ class information gain ’, it does not resolve the domain misalignment problem between the source and target domain.
Because the target labels are not available during training, we try to keep the overall distribution of the target data as close to that of the source data as possible.
Since the source data are evenly split, we guide the algorithm to minimize the information gain between the source and target domain, which encourages even split of the target data also.
The CoBRF combines the ideas, minimizing the ‘ domain information gain ’ between source and target data for the domain alignment while keeping the class information gain to be maximized.
Note that the domain alignment term is the same as the negative information gain of the binary domain labels (source/target).
Thus, the CoBRF can be seen as an example of adversarial learning, as it considers the domain information gain in an adversarial manner compared to the conventional objective function of the random forest.
We summarize the main contributions as three-fold.
• We introduce the collaborative training algorithm based balanced random forest (CoBRF) using the discriminative and even node split function.
Linear SVM with binary pseudo labeling is used to ﬁnd the discriminative hyperplane and the even split ensures the decision tree to be balanced.
• We also adopt the adversarial learning of domain information gain to align the source and target data distribution.
To align two domains, the information gain between the source and target data is minimized, which learns the common data distribution of both the (unlabeled) target domain and the source domain data.
• We perform an extensive evaluation of the domain adaptation to show the performance of the proposed method according to various challenging evaluation protocols.
Speciﬁcally, it is compared to the baseline and state-of-the-art methods using noisy and small training data, and with open-set domain adaptation protocols.
In both cases we observe signiﬁcant performance improvements.
R ELATED W ORK 2. D OMAIN A DAPTATION
Recently adversarial learning has been one of the dominant approaches in domain adaptation with deep neural networks.
The gradient reversal layer Ganin & Lempitsky (2015) is introduced to train the networks so that the discrimination of source and target domains is penalized.
It improves the classiﬁcation performance compared to the networks learned only with the source data.
Tzeng et al.
(2017) suggest the domain adaptation framework based on the discriminative network learning, which assigns individual weights to the source and target domains.
In training the networks, they also consider the adversarial weight update to align the domains.
Several other domain adaptation papers in adversarial learning using conditional learning Long et al.
(2018), domain-symmetric Zhang et al.
(2019), and collaborative Zhang et al.
(2018b) methods have been introduced.
Also, in Tzeng et al.
(2014); Long et al. (2015; 2016), maximum mean discrepancy (MMD)-based methods have been studied.
Tzeng et al.
(2014) propose the domain confusion loss to improve domain distribution alignment.
Long et al.
(2015) introduce the task-speciﬁc embedding and multiple kernel approach along with MMD to decrease the domain discrepancy.
The residual transfer module presented in Long et al.
(2016) associates the classiﬁcation ability of the source and target domain.
MMD is further extended to multiple domain alignment in the joint adaptation networks (JAN) Long et al.
(2017) using adversarial learning.
The generative adversarial networks Radford et al.
(2015) are adopted in many domain adaptation methods Liu & Tuzel (2016);
Sankaranarayanan et al.
; Volpi et al.
CoGAN proposed by Liu & Tuzel (2016) learns the joint distribution of multiple domains without corresponding image pairs.
Sankaranarayanan et al.
(2018) propose the combined adversarial and discriminative learning method using the generator and discriminator of GAN.
2. E VALUATION P ROTOCOLS IN D OMAIN A DAPTATION
Recently, many challenging protocols are introduced to evaluate the domain adaptation in realistic settings.
Regarding domain generalization on deep neural networks Li et al.
Balaji et al.
(2018), they divide multiple domain data into training and test set, then use the leave-one-domain-out scheme for evaluation.
The domain adaptation on the partially overlapping source and target domains is presented in Zhang et al.
(2018a); Cao et al.
Multiple sources and target domains are mixed into the source or target domains in Zhao et al.
; Mancini et al.
; Hoffman et al.
The adaptable model is aimed to be learned using the distribution to the multiple domains of the mixed set.
Recently, several works Saito et al.
; Panareda Busto & Gall (2017); Tan et al.
(2019) address the open set domain adaptation.
They assume that there exist unknown and partially overlapped known classes between domains.
On the other hand, the domain adaptation methods under small training data Hong et al.
(2017) and the noisy data Shu et al.
(2019) are studied to address the real-world condition.
Hong et al.
(2017) use single training data per person, and Shu et al.
(2019) artiﬁcially corrupt the class labels or features of the source domain for the robustness evaluation.
These protocols are challenging as they pose difﬁcult problems of overﬁtting, class misalignment, noisy, lack of training data, and little overlap.
2. R ANDOM F OREST AS AN E NSEMBLE L EARNING M ETHOD
The ensemble of multiple learners has widely been used to avoid the overﬁtting problem Singh et al.
; Han et al.
(2017; 2016); Pi et al.
Singh et al.
(2016) introduce the regularization method for network learning, which works with a variety set of network architectures and performs better than the existing regularization methods ( i .
e .dropout).
Branchout Han et al.
(2017) is devised for layer-level regularization in visual tracking, where multiple branches of fully connected layers are randomly selected in training.
Random forest Breiman (2001) combines multiple random decision trees to build robust classiﬁer or regressor.
Random forests have been applied to many applications such as object tracking Zhang et al.
(2017), feature point detection Lindner et al.
(2014), and speech recognition Black & Muthukumar (2015), to name a few.
However, it should be emphasized that the most important beneﬁt is the mitigation of overﬁtting by ensembling multiple decision trees.
As noticed in the literature Wyner et al.
; Gomes et al.
(2017), the random forests tend not to propagate severe overﬁtting error even with a large number of trees.
There have been many recent works to improve the performance of random forests: Dheenadayalan et al.
(2016) proposes pruning nodes for efﬁcient learning, Ristin et al.
(2015a)
presents incremental modeling for large scale recognition, and Probst & Boulesteix (2017) investigates how to tune the number of trees.
SVM Yao et al.
; Ristin et al.
(2015b) or random projection Bosch et al.
(2007); Bossard et al.
(2014) is often used as the binary classiﬁer for better node split.
Training balanced decision trees has been also an important topic Bosch et al.
; Bossard et al.
Yao et al.
(2011); Lei et al.
; Ristin et al.
(2015b).
They split a node into child nodes by the binary classiﬁer, which is trained by evenly-divided training data in the node.
We argue that training balanced random forests helps to alleviate the overﬁtting problem since balancing random decision trees avoids the biased distribution in the speciﬁc domain but prefers the common representation to any domains.
Hence, we introduce the learning algorithm that enforces the even-split constraint by shifting the hyperplane(Sec. 3.1) for balanced random forests.
Although there have been studies of the balanced training of random forests, we provide elaborate training process of balanced random forests to learn common representations for the domain adaptation task.
The effectiveness of the balanced random forests is shown by extensive domain adaptation experiments.
[Figure] [Figure]
P ROPOSED M ETHOD
In this section, we ﬁrst explain the limitation of the conventional random forests for the domain adaptation task, and then we introduce the even node split function in Sec.3.1 and the domain information gain for selecting the domain-aligned split function in Sec. 3.2.
3. E VEN C ONSTRAINED R ANDOM F OREST L EARNING
A random forest consists of multiple random decision trees, whose nodes learn a binary classiﬁer for the randomly-selected subset of features to maximize the information gain (IG).
We abuse the term node for the training data in the node interchangeably.
The entropy of a node n is deﬁned as
[Equation]
where C ( n ) represents the set of classes of the data in n , and p c ( n ) is the probability of class c in n (i.e., the data count of class c divided by | n | ).
Then the information gain for a node n with the left and right child nodes is deﬁned as
[Equation]
Conventionally, the simple split functions that compares only a couple of feature values are used, but recently more elaborate split functions using the linear classiﬁers are used Yao et al.
; Ristin et al.
(2015b).
The hyperplane split function for a node n is written as
[Equation]
where ψ n ( · ) is the sub-feature selection function and w n and k n are the hyperplane parameters either randomly set or learned by a linear support vector machine Cortes & Vapnik (1995).
The hyperplane with the largest information gain is the most discriminative classiﬁer at the given node, but for the entire decision tree and the random forest it may not be the best option, because it causes the learned trees to be skewed and not well balanced (Fig. 1a).
We propose to add a hard constraint of equal-size in splitting the node to get more balanced trees.
The detailed learning process is as follows.
For the SVM to build a binary classiﬁer, the classes in the node are randomly assigned to binary pseudo labels, and the training data for each class are assigned to the corresponding pseudo label.
As the data sizes of the pseudo labels will be different, we randomly erase the data in the larger pseudo class to match the sizes.
Then the base hyperplane ( w n and k n ) is computed to classify the binary pseudo labels.
Still, the split of the training data by the hyperplane is not equal-sized; thus we update the bias k n of the hyperplane so that the data size on each side is equal or differs at most by one ( || n left |−| n right || ≤ 1 ).
Geometrically this process is moving the hyperplane along the normal vector w n , so that it is placed at the even split of the data projected onto the normal direction (Fig. 2a,2b).
Among the estimated hyperplanes from randomly selected sub-features, the one that maximizes the information gain
IG C ( n ) is chosen as the node split function.
To build a decision tree, like the conventional random forest, the node split is repeatedly applied until the maximum depth is reached or too few data are left in the node (Fig. 1b).
Inherently the proposed split method creates balanced trees, and for the same depth, the number of nodes is much larger than that of the conventional random forest.
We argue that having more (leaf) nodes in the decision tree has advantages in domain adaptation tasks.
The conventional split function is locally optimal, but because of that, it is more susceptible to overﬁtting by committing too early, and eventually, it decreases the discriminative power of the entire random forest.
In the balanced trees, the data sizes in the leaf nodes are almost the same; thus, they represent local data distribution more faithfully.
The even-size constraint can be thought of as a regularization in learning decision trees.
The experimental results of the ablation study in Sec.
4.2 supports this argument.
3. C OLLABORIVE L EARNING OF R ANDOM F ORESTS
Balanced data distribution is a big advantage in domain adaptation.
However, as it does not use the unlabeled target data for learning, it still does not correctly align the data distribution of the source and target domain.
In other words, the distribution of the target data also needs to be considered in building a random forest.
We propose a new collaborative measure for selecting the split function that considers both the conventional IG and the domain distribution of the source and target data together.
The collaborative information gain (co-IG) is deﬁned as
[Equation]
where λ is a user parameter, n s is the labeled training data (in the source domain), and D is the binary domain label { source, target } representing the domain that the data belongs to.
More speciﬁcally,
IG D ( n ) is the information gain on the domain distribution, when the data labels are either source or target, disregarding the classes in the source domain.
The CoBRF chooses the hyperplane that maximizes co-IG when splitting the nodes.
Note that the IG on the domain distribution, IG D ( n ) , is subtracted in Eq. 4, to ensure that we prefer even distribution of the source and target data in the child nodes.
IG D ( n ) is minimized when both
[Figure]
source and target data are evenly split into the children, as it maximizes the entropy of the children (Eq. 2).
Thus IG D ( n ) in CoBRF collaboratively enforce the even split of target data also.
Fig.
2c illustrates the effect of co-IG compared to conventional IG in split function selection.
The yellow line has higher IG as it segments the source data (colored) better, but co-IG also considers the split of target data (gray).
Although the blue line has lower IG than the blue, it separates the target data more evenly; thus, the blue line is chosen as the split function.
The resulting decision trees by
CoBRF are shown in Fig.
The co-IG is closely related to the adversarial learning of the network backpropagation Long et al.
; Ganin et al. (2016).
In this framework, IG C and IG D can be thought of as the classiﬁcation and adversarial domain alignment, respectively.
Thanks to the domain alignment term (co-IG), the
CoBRF learns the robust models even with very noisy or small training data without overﬁtting.
We validate the proposed methods from the moderate challenging condition such as 40% noise data to the very severe condition such as 80% noise or merely 10% training data.
Further, we evaluate the open set domain adaptation, which has received attention in recent years, in the following section.
E XPERIMENTAL R ESULTS 4. E XPERIMENTAL S ETTING
We use three domain adaptation datasets such as Ofﬁce-31 Saenko et al.
ImageCLEF-DAand Ofﬁce-Home Venkateswara et al.
We evaluate algorithms using three challenging protocols: noisy, small training data, and weakly supervised open set domain adaptation.
To train the
CoBRF, we use 100 trees with a maximum depth of 8.
When there is no training data fallen on a node, we prune the tree at that node.
The number of randomly selected feature dimension for the
SVM training is set to 250.
The input feature of SVM is normalized for the stable learning of the hyperplane.
We repeat the SVM training 15 times to select the optimal split in each node.
Due to the space limitation, only representative results are shown in this section.
Refer to the appendix for detailed information of the datasets, metric, and full experimental results.
4. A BLATION S TUDY
We evaluate the effect of components in the CoBRF proposed in this paper.
The CoBRF uses the balanced pseudo labeling (mid_pseudo), and the hyperplane shift (h_shift) for even data split.
As binary labeling is necessary for hyperplane computation, the pseudo method uses randomly-assigned binary pseudo labels without removing the data to make label sizes equal. Therefore the four
[Table] [Table]
[Table] [Table]
combinations of (pseudo, mid_pseudo) × h_shift are tested with Ofﬁce-31.
The baseline is (pseudo + no_h_shift).
As shown in Table 1 and 8 of appendix, both balancing the pseudo labels and enforcing even splits by translating hyperplanes improve the performance.
Table 2 shows the effect of the parameter λ in co-IG formulation (Eq. 4).
It conﬁrms that optimizing for the cobalanced distribution helps the alignment of domain distributions.
Table 3 also represents the effect of hyperparameters in training CoBRF.
The accuracy of the maximum depth 8 withdecision trees shows the best result.
4. N OISY D ATA
In this experiment, the training labels of the speciﬁed portion of the source domain are randomly changed for the noise condition, which is also referred to as the label corruption in Shu et al.
Corruption levels are set to 40 , 60 and 80% of the source domain (refer to the supplementary material for full experimental results).
We conduct noisy conditions for the Ofﬁce-31, ImageCLEF-DA and Ofﬁce-Home datasets in Tableand 5.
We test DAN Long et al.
JAN Long et al.
(2017), and CDAN+E Long et al.
[Table] [Table]
[Table]
algorithms 2 on the same noisy condition for comparison.
Table 5 shows the result of 40% noisy training data for Ofﬁce-31 with ResNet-50.
The proposed CoBRF outperforms all other algorithms in average accuracy.
The result conﬁrms that the CoBRF improves the performance in most settings, and interestingly, Table 4 shows the more severe the noise is, the larger the performance improvement gets.
4. S MALL T RAINING D ATA
In this experiment, we use only 10% of training samples to evaluate the performance against overﬁtting.
We perform the experiments on Ofﬁce-31, Ofﬁce-Home, and ImageCLEF-DA datasets with
ResNet-50.
The result of Table 4 shows the CoBRF achieves favorable performance compared to the other algorithms.
Full experimental results are presented in the appendix.
4. O PEN S ET E XPERIMENTS
We perform two open set evaluation protocols proposed in Saito et al.
Tan et al.
OpenSet1:
The ﬁrst open set protocol Saito et al.
(2018) uses 11 classes (10 known and 1 unknown) of the Ofﬁce-31 dataset.
The labels from 1 to 10 of both source and target domains are marked as the known class, and all data with label 21 ∼ 31 in the target domain are used as one unknown class.
According to Saito et al.
the unknown class of the source data is not used in training, and the unknown class of the target data is classiﬁed by thresholding the class probability.
The thresholding value is set to 0.35.
Table 6 shows the result of CoBRF as well as the state-of-the-art methods.
The
CoBRF achieves the best performance among all algorithms on Ofﬁce-
It also demonstrates the effectiveness of the proposed method under the challenging adaptation condition.
OpenSet2: Recently, another open set protocol is proposed in Tan et al.
(2019), which uses partially overlapping known classes between the source and target domain.
Each domain has 5 known-andcommon classes, 5 known-but-different classes, and 1 unknown class for all other training data, thus in total there are 15 known and 1 unknown classes.
First, according to Tan et al.
(2019), 3 samples per class per domain and 9 samples in the unknown class per domain are used in training.
Hence the total number of training samples is (3 samples × 10 classes/domain + 9 samples_in_unknown)
×domains = 78.
All other algorithms and CoBRF results in Table 7 are using this protocol.
Additionally, we evaluate more challenging setup, where the training data are sampled regardless of the domain, i.e., the data in common classes (including unknown) are merged before being sampled.
In this case, 3 samples × 15 classes + 9 samples_in_unknown = 54 in total are used.
The results in CoBRF* rows are acquired in this setup.
We conﬁrm that CoBRF works well compared to state-of-the-art methods under the OpenSet2 and more challenging condition.
C ONCLUSION
We propose a novel cobalanced random forest (CoBRF) algorithm for challenging conditions and open set protocols.
The CoBRF enhances the discriminative ability of the random forest by building balanced decision trees by the even split.
The proposed CoBRF algorithm also employs the adversarial learning for domain alignment and beneﬁts the effectiveness against the overﬁtting to the labeled source data.
We extensively evaluate the proposed algorithms using challenging experimental protocols and demonstrate its superior performance over the baseline and state-of-the-art methods.
Appendix A D ATASET AND M ETRIC
We use three domain adaptation datasets for the experiments.
Ofﬁce-31 is the domain adaptation dataset, which consists of three domains, Amazon (A), Webcam (W) and DSLR (D).
The dataset has 4,652 images with 31 categories, and we evaluate all combinations of the domains transfer following the previous work.
ImageCLEF-DA includes 12 classes in each domain with 50 images per category per domain.
The three domains of Caltech-256 (C), ImageNet ILSVRC 2012(I) and Pascal VOC 2012 (P) are used for evaluation.
Ofﬁce-Home with 15,500 images and 65 categories is a larger dataset than ofﬁce-31.
We use four domains of Artistic (A), Clip Art (C), Product (P) and Real-World (R) for evaluation.
We follows the standard metric for the experiments.
The averaged accuracy over the classes Saito et al.
Tan et al.
(2019) is used to OpenSet1 and OpenSet2, and for other experiments the classiﬁcation accuracy is used as the metric.
B A BLATION S TUDY
The full tables of ablation study in the manuscript are presented.
[Table]
λ
Domain adaptation
A → D A → W D → A
D → W
W
→ A
W → D
Average 76.
C N OISY D ATA
The full tables of noisy protocol in the manuscript are presented.
[Table] [Table]
[Table]
D S MALL T RAINING D ATA
The full tables of small training data protocol in the manuscript are presented.
[Table]
[Table] [Table]