MetaTeacher: Coordinating Multi-Model Domain Adaptation for Medical Image Classification
Abstract
In medical image analysis, we often need to build an image recognition system for a target scenario with the access to small labeled data and abundant unlabeled data, as well as multiple related models pretrained on different source scenarios.
This presents the combined challenges of multi-source-free domain adaptation and semi-supervised learning simultaneously .
However, both problems are typically studied independently in the literature, and how to effectively combine existing methods is non-trivial in design.
In this work, we introduce a novel MetaTeacher framework with three key components: (1) A learnable coordinating scheme for adaptive domain adaptation of individual source models, (2) A mutual feedback mechanism between the target model and source models for more coherent learning, and
(3) A semi-supervised bilevel optimization algorithm for consistently organizing the adaption of source models and the learning of target model.
It aims to leverage the knowledge of source models adaptively whilst maximize their complementary benefits collectively to counter the challenge of limited supervision.
Extensive experiments on five chest x-ray image datasets show that our method outperforms clearly all the state-of-the-art alternatives. The code is available at https://github.com/wongzbb/metateacher.
Introduction
Despite the great stride made by existing deep learning methods on medical image classification results [32, 53, 67], their performances often degrade drastically when applied to a new unseen scenario.
This is mainly due to the domain shift challenge between the training and test data, caused by different environments, different instruments, and different acquisition protocols.
Unlike natural images, annotating medical images requires special clinical expertise.
It is hence more difficult to obtain large-scale medical image datasets with high-quality labels at every single scenario.
Domain adaptation is a feasible solution, but comes with several limitations.
Firstly, medical data is often under strict privacy and license constraints.
That means the source domain data is usually inaccessible during domain adaptation.
Secondly, medical data is typically multi-labeled which means that there are multiple labels for a sample, and the multiple categories are not mutually exclusive.
It has more prominent different characteristics in different scenarios.
Considering these practical constraints, we propose a new Semi-supervised Multi-source-free Domain Adaptation (SMDA) problem setting in the context of medical image classification.
Our proposed setting has three key conditions: (1) There are multiple source domain models trained on respective multi-label medical image datasets; (2) All The the source domain data is inaccessible for adaptation; and (3) The target domain data has only a small number of labelled samples along with abundant unlabeled data.
In medical image classification, there are limited domain adaptation works, with a need of accessing the source domain data [5,19,24,34,45,55,57,62].
Further, they usually consider a single source domain.
On the other hand, for employing multiple source domains, existing Multi-Source Domain
Adaptation (MSDA) methods typically learn a common feature space for all source and target domains [58] or use ensemble methods combined with source classifiers [8].
However, all of these
MSDA methods require access to the source domain data.
Regarding multi-label medical image classification, there exists a solution which extends the standard classifier network by conditional adversarial discriminator networks [46].
But it is still not source-free.
Indeed, there have been extensive study on Source-Free Domain Adaptation (SFDA) [35,64].
However, they are not directly applicable to our problem.
Firstly, most of them assume a single source domain [35,64].
Using a
SFDA method to transfer each source domain model to the target domain separately and average their predictions, this strategy cannot reveal the complementary information between different source domains.
Secondly, the source model is often domain biased.
Different hospitals are featured with different populations, leading to a situation that the source datasets focus on a specific set of class labels.
The existing SFDA methods can not assess the credibility of a source domain model with different labels.
To address the above SMDA’s limitations, employing knowledge distillation from multi-source models to the target domain can be considered [18,42,65,69,70].
This forms a multi-teacher and one-student scheme.
In our problem setting, a few labels of the target domain are provided to judge the credibility of multi-source models in different labels.
In reality, it is common to exploit a few labeled data in the target domain.
Recent works [25, 29, 50, 51] have shown that a few labeled data from the target domain can significantly improve the performance of the model.
Inspired by meta-learning approaches [40, 47, 49], we consider a bilevel optimization strategy to update both the teachers and students.
This is because different models vary in reliability and there is a need for optimizing the update direction for each source model.
This offers an opportunity of leveraging the complementary and collaboration of different source models during model optimization, critical for solving the low-supervision challenge.
Based on the above analysis and consideration, we propose a novel framework, namely MetaTeacher .
Specifically, it is based on multi-teacher and one-student models.
Each teacher model is pre-trained on a specific labeled source data.
The student model is initialized by a randomly chosen teacher.
In order to provide different update directions for multiple teachers, a coordinating weight learning method is proposed to determine the contribution of each teacher for each target sample.
In addition to knowledge transfer from multiple teachers, when adapting a specific teacher model, we also explore the feedback from the student and other teachers in a semi-supervised meta learning manner [16,47].
Unlike the previous MSDA approaches, MetaTeacher can adapt each teacher in different directions according to the learned coordinating weight.
This enables us to fully use different characteristics of source models, whilst avoiding the problem of insufficient training samples for multi-label classification to some extent.
Our contributions are summarized as follows: (1) We propose a new problem setting, i.e., semisupervised multi-source-free domain adaptation for multi-label medical image classification.
To our best knowledge, our work is the first attempt at multi-source-free and semi-supervised domain adaptation in the field of transfer learning.
(2) A novel framework, MetaTeacher, based on a multiteacher and one-student scheme is introduced to solve the proposed SMDA problem.
A mutual feedback mechanism is designed based on meta-learning between the target model and the source models for more coherent learning and adaptation.
The knowledge from multiple source models are sufficiently leveraged.
A coordinating weight learning method is derived for dynamically revealing the performance differences of different source models over different classes.
It is integrated with the semi-supervised bilevel optimization algorithm for consistently updating the teacher and student models.
Extensive experiments on five well-known chest radiography datasets show that our approach outperforms state-of-the-art alternatives clearly, along with in-depth ablation studies for verifying the design of our model components.
Relate Works
Unsupervised domain adaptation for medical image classification.
There exist shallow UDA and deep UDA approaches in the literature.
Shallow UDA approach adapts two routes, i.e., source domain instance weighting [55,57] and feature transformation [24,34].
All of these methods need to access source domain data.
Similarly, there are also two routes for deep UDA approach.
They are domain alignment based [19,62] and pseudo-labeling based [5].
The first strategy solves the UDA problem by minimizing the domain difference between the source domain and target domain, and is currently the most popular method.
Gao et al.
[19] used the central moment difference matching to perform adaptation of classifying brain MRI data.
The second strategy generates dummy data to retrain target model.
For multi-label medical image classification, there exists a work based on domain alignment with a multi-label regularization term [46].
Bermúdez Chacón et al.
[5] used the normalized cross-correlation to generate soft labels for the target domain.
The above UDA methods do not update the source domain model, and they are all based on single-source domain.
However, the situation of multi-source domains is very common in practical situations.
Source-free domain adaptation. Source-free domain adaptation methods can be roughly divided into two routes, i.e., generative approach [27,28,33,63] and pseudo-label approach [4,26,35,56].
The generation approach generates target-style training samples to train the prediction model.
Since learning to generate features is difficult, this approach is extremely limited.
The pseudo-label approach generates pseudo-labels through the source domain model, which is simple and general and has recently achieved good results in the machine learning community.
The research of source-free domain adaptation in the medical image analysis field mainly focuses on image segmentation.
Bateson et al.
[4] maximized the mutual information between the target images and their label predictions to perform spine, prostate and cardiac segmentation.
Vibashan et al.
[56] implemented source-free domain adaptive image segmentation by generating pseudo-labels and applied self-training methods for task-specific representation.
These works are all conducted in the single-source domain case.
Currently, the research on multi-source-free domain adaptation is extremely limited, and most of the works adapt the method of generating trusted pseudo-labels [1,14].
Multi-source domain adaptation for medical image classification.
In machine learning community, MSDA works mainly have two strategies, i.e. distribution alignment [43,74] and adversarial learning [61,71,72].
The first strategy computes the statistical discrepancy between multi-source domains and target domain, and then combines all predictions.
The second strategy trains a domain discriminator and forces the feature extraction network to learn domain-invariant features to confuse the domain discriminator.
For medical image classification, there only exist several shallow DA models.
Wang et al.
[58] proposed to map multiple source and target data to a common latent space for autism spectrum disorder classification.
Cheng et al.
[8] constructed a multi-domain transfer classifier for the early diagnosis of Alzheimer’s disease.
All of these strategies require to access source domain data and are not suitable for solving the proposed SMDA problem.
To the best of our knowledge, current teacher-student domain adaptation methods in the medical and machine learning communities only consider the single-source domain case.
When extended to the multi-source domain, it will face a challenging multi-objective optimization problem [10,41].
Semi-supervised domain adaptation (SSDA).
Our problem is also related to SSDA which assumes a small number of labeled samples in the target domain.
Compared to UDA, using a few labeled samples of the target domain allows to further achieve better domain alignment [31,44,66].
Due to the shift of domain distribution, directly applying classical semi-supervised learning methods to the SSDA problem will lead to sub-optimal performance.
Representative SSDA works are based on subspace learning [44, 66], entropy minimization [20, 50], label smoothing [13, 48] and active learning [48, 52].
However, all of these methods assume a single source domain with the source domain data accessible.
Unlike these works, our method incorporates meta-learning and uses the performance on the labeled target data as a feedback signal.
Teacher-student domain adaptation models in medical image analysis.
Usually, teacher-student domain adaptation model proposes multiple consistencies to solve UDA problem.
To the best of our knowledge, teacher-student based domain adaptation methods have received little attention on medical image analysis.
Perone et al.
[45] proposed a semi-supervised learning based UDA method for medical image segmentation, which minimizes the consistency loss of the predicted results between the student model and the teacher model for unlabeled samples in the target domain during the training process.
The network is updated by the exponential moving average of the student
T a r g b e e d d m a ll )
T a r g e t un l a b e l e d d a t a ( l a r g e )
C oo r d i n a ti ng W e
i gh t
C oo r d i n a ti ng W e
i gh t
T eac h e r T eac h e r n
S t
ud e n t
T eac h e r
T eac h e r n
P r ob a b iliti e s
F e a t u r e
M a
pp i
ng M a
pp i ng
[Figure]
[Figure]
) B il e v e l O
p
ti m i za ti on
S t
ud e n t
F ea t u r e
E
x t r ac t o r & C l a ss i f
i e r
I npu t
l a b e l e d d a t a
I npu t
un l a b e l e d d a t a (cid:17)(cid:17)(cid:17) (cid:17)(cid:17)(cid:17) network weights (mean-teacher [54]). The method is experimentally performed on the SCAM (Spinal Cord Anatomy MR Image) dataset to demonstrate its effectiveness. There are several approaches on teacher-student domain adaptation in the field of machine learning.
French et al.
[17] made some modifications to the mean-teacher scheme for the challenging domain adaptation of natural image classification.
Cai et al. [7] proposed multiple consistency regulations to solve cross-domain detection problem. Deng et al. [12] combined the idea of feature alignment and data augmentation based on mean-teacher scheme.
These methods all assume single-source domain, and to our knowledge, there is currently no work on multi-source teacher-student domain adaptation.
Additionally, the mean-teacher approach does not sufficiently use the feedback signal from the target domain, so the performance improvement is limited.
Methodology
Problem statement.
Suppose D T
= { ( X tL , Y tL ) , X tU } where Y tL denotes label annotations for a small amount of target domain samples X tL and X tU for target domain samples without any label annotations.
The dimension of label vector is m .
D S i
= (cid:8)(cid:0) X iL , Y iL (cid:1)(cid:9) where Y iL denotes label annotations for i -th source domain samples X iL .
For the proposed semi-supervised multi-source-free domain adaptation problem, when the pretrained source classifiers f T i is applied to the target domain, the source dataset D S
i is not accessible for i = 1 , · · · , n .
Given the source classifiers f T i for i = 1 , · · · , n and the target data D T , the objective is to find a target-domain mapping f S :
X tU → Y tU where Y t U denotes the predicted labels for target domain samples X t U .
Overview.
As shown in Fig.1, our framework is based on a multi-teacher and one-student scheme .
First, multiple teacher models are pretrained according to each source domain, and then the student model is initialed using a randomly chosen teacher model.
They are all composed of a feature extractor (e.g., Resnet50 [21]) and a multi-label classifier.
The classifier consists of a fully connected layer, where the input is an one-dimensional expanded feature, and the output is the probability of each label.
The objective function is the error loss between the predicted output and the ground truth.
Compared with traditional teacher-student model, our method are featured with two unique parts: (1) Coordinating weight learning ; (2) Bilevel optimization .
For the first part, a mapping is trained based on labeled target domain samples, which fuses the multi-teacher predictions adaptively for each target sample.
This mapping will be used in the second part.
In the initial iteration, the mapping and student model are trained based on labeled target samples.
In the subsequent iterations, this part will only optimize the mapping while the student model will be updated by bilevel optimization.
In the bilevel optimization part, the student and teacher models are updated alternately in a meta-learning manner.
Specifically, for an unlabeled target sample, a coordinating weight is generated, which provides optimization direction for each teacher model.
Finally, these two parts will be iteratively undated until convergence.
3.1 Coordinating Weight Learning
As mentioned earlier, the teacher models are trained on different source domain data.
Due to different distributions, they often present different characteristics.
Therefore, for a target domain sample, the classification probability of each teacher model could be inconsistent.
When we want to optimize a teacher model based on the target domain samples, the optimization direction of each teacher model should be different.
So it is necessary to obtain the contribution weight of each teacher model to the final classification results.
We call this coordinating weight .
Fortunately, we can obtain the weight mapping with the labeled samples in the target domain.
As shown in Fig.1(a), for obtaining the coordinating weight, we first input the labeled target sample x tl into the student network, and get the output B = g ( x tl ) from feature extraction network g , where
B ∈ R c × h × w , with c , h , and w the number of channels, height, and width respectively.
Then, we perform a maximum pool operation on the feature map B to get ψ ∈ R 1 × c which retains the most important information of each channel.
Our mapping consists of two learnable variables µ and ν , where µ ∈ R n × 1 , ν ∈ R c × m .
Then, we define a mapping ϕ =
µψν ∈ R n × m for the target sample x tl .
After normalizing, we get the coordinating weight matrix W where
[Equation]
Suppose for the sample x tl , the predictions of all teachers are formed as a matrix P ∈ R n × m . By taking the Hadamard product between the teacher predictions and the coordinating weight matrix, we obtain the fused prediction as the following,
[Equation] [Equation]
where Sum ( · ) means adding by rows.
Denoting ¯ y sl = f S
( x tl ; θ S ) as the student prediction on the target sample x tl , we train the weight mapping and initialize student network using the following loss,
[Equation]
where L (¯ y sl , y l )
= 1 m (cid:80) mi =
[ y l,i log (¯ y sl,i )
+ (1 − y l,i ) log (1 − ¯
y sl,i )] represents the BCE (Binary
Cross Entropy) loss, y l is the ground truth, θ S is the parameter of student network.
L KL (¯ y tl , ¯ y sl ) = (cid:80) mi =1 ¯ y tl,i log(¯ y tl,i / ¯ y sl,i ) represents the KL (Kullback-Leibler divergence) loss which measures the distribution difference between the fused teacher prediction and student prediction.
α and β are two balance parameters.
Remark.
The mapping ϕ generates coordinating weight with Eq.(1). It not only reveals the complementarity of different teachers on different instances, but also, more interestingly, participates in the derivation of the update formula of teacher models in the bilevel optimization process (see Appendix), providing a reference for the update direction of different teachers.
3.2 Bilevel Optimization
The bilevel optimization problem [6,9] was first proposed in the field of game theory.
It includes an upper-level optimization task and a lower-level optimization task , where the former contains the latter as a constraint.
Here, the upper-level optimization task (student) provides feedback signals to the lower-level optimization tasks (teachers) through the performance on labeled data and the coordinating weight mapping.
For an unlabeled target sample x tu , suppose the pseudo-label based on the learned coordinating weight mapping ϕ from multi-teachers Eq.(2) is ¯ y tu and the corresponding coordinating weight matrix is W u , we can define a loss function Γ u as follows,
[Equation] [Equation]
where ¯ y s u = f S ( x t u ; θ S ) , θ T i is the parameter of the i -th teacher network.
Similarly, a loss function Γ l ( θ T 1 , · · · , θ T n , θ S ) =
L ( y l , ¯ y sl ) is defined for a labeled target samples x tl .
In the bilevel optimization task, updating θ S is the upper-level optimization task objective, while updating θ T 1 , · · · , θ T n is the lower-level optimization task objective.
The upper-level optimization task and the lower-level optimization task are mutually constrained.
To reach the lower-level optimization task objective, the performance of the upper-level optimization task objective on the labeled target data is utilized as feedback signal .
We formulate the objective function in lower-level optimization task as the following,
[Equation]
Eq.(5) cannot be optimized simply by gradient descent algorithm, because the teacher’s parameters can not be updated until θ S reaches the optimum.
To overcome this issue, we resort to the idea of meta-learning [16,38,47] by making a one-step approximation of the problem,
[Equation]
where η S is the learning rate of the student network.
Substituting Eq.
(6) into Eq.
(5), we obtain a new optimization objective function
[Equation]
By optimizing Eq.
(7) (see Appendix), we get the following update rules,
[Equation]
for i = 1 , · · · , n , where θ ′ S and θ ′ T
i are the updated parameters corresponding to the student and teachers respectively.
¯ y iu =
f T i ( x tu ; θ T i ) · W iu with W iu the i th-row coordinating weight vector of W u w.r.t the i -th teacher.
We obtain the pseudo label ˜ y iu by binarizing ¯ y iu {+as: ˜ y iu,j = 
+}
when ¯ y iu,j < 0 .and
˜
y iu,j = 1 for the other cases.
Additionally, in order to prevent optimizing teachers in the same direction, the predictions of the updated multiple teachers should be as far away from each other as possible.
To that end, we define a divergence loss as follows,
[Equation]
where B T i ( x u ; θ T i ) represents the max-pooled results of the output feature map of the i -th teacher network.
Here, we apply a max-pooling operation to the output features of multiple teachers and calculate the distance with L 2 norm.
By requiring these feature maps to be far away each other, the optimization direction of teachers will be effectively adjusted.
Finally, we update the i -th teacher network by the following rule,
[Equation]
) where γ is a hyperparameter.
Remark.
Eq.(11) reveals that the update direction of θ T i is determined by three factors: (1) Coordinating weight confuses the feedback signals from different teachers; (2) Student network parameters provide the feedback signals and generate coordinating weight; (3) Diversity constraint emphasizes the characteristic of different teacher networks.
Interestingly, these three factors change over time during the meta-learning process.
In addition to alternating updates of the student and teacher models, we also update the mapping periodically. 3.3 Optimization Process
The optimization process alternates between upper-level optimization and lower-level optimization.
Since the coordinating weight mapping is fixed in the process of bilevel optimization, with the progress of meta-learning, the coordinating weight W u can not reflect the internal relationship between the source domains and the target domain gradually.
Therefore, we use the learned teachers to update the mapping at regular intervals.
The training process is summarized in Algorithm 1.
Experiments
Datasets.
Five publicly available chest x-ray datasets are used to construct our multi-domain adaptation scenarios.
NIH-CXR14 [59] is a large public dataset of chest x-ray, which contains 108, Require: Student network parameters S (0) , teacher network parameters T (0) ∼ T (0) n , labeled data ( x tl , y l ) , unlabeled data ( x tu ) , hyperparameters α, β, γ , mapping updating interval T . Ensure: Optimized student model S ( N ) .
{+1: function M ETA T EACHER ( S (0) , T (0) ∼ T (0) n , α, β, γ, T ) 2: S (0) , mapping ← Coordinating Weight Learning 3: for t = 
 → N − 1 do Upper-level optimization: 4:+}
Compute gradient ∇ θ S ( t ) L u 5:
Update the student: θ S ( t +1) ← θ S ( t ) − η S ∇ θ St L u ▷ Eq.(8)
Lower-level optimization : 6:
Compute gradient ∇ θ S ( t +1)
L l 7: for all T ( t ) 1 ∼ T ( t ) n do 8:
Compute gradient ∇ θ T ( t ) i L (cid:0)
¯ y iu , ˜ y iu (cid:1) 9:
Compute gradient ∇ θ T ( t ) i
L D ▷ Eq.(10) 10:
Update the i -th teacher: θ T ( t +1)
i ← θ T ( t ) i − η T i · ▷ Eq.(11) ([( ∇ θ S ( t +1) Γ l ) T · ∇ θ S ( t ) Γ u ] T · ∇ θ T ( t )
i
L (cid:0)
¯ y iu , ˜ y iu (cid:1) + γ ∇ θ T ( t ) i L D ) 11: end for 12:
if t % T = 0 then 13: mapping ← Coordinating Weight Learning 14: end if 15: end for 16: return S ( N ) 17: end function front view x-ray images of 32,717 patients collected from NIH Clinical Center, with a total ofdisease labels.
MIMIC-CXR [23] contains 377,110 images and text reports, corresponding to 227, radiological studies conducted by Beth Israel Deaconess Medical Center in Boston, Massachusetts.
CheXpert [22] consists of 224,316 chest x-ray of 65,240 patients.
The dataset collected chest x-ray examinations and related radiology reports performed at inpatient and outpatient centers at Stanford
Hospital from October 2002 to July 2017.
Open-i [11] is collected by Indiana University Hospital through the network from open source literature and biomedical image collection.
It containsradiology reports, corresponding to 7470 frontal and lateral chest films.
To be consistent with other datasets, we filter out the side chest x-ray in Open-I, leaving only 3955 frontal images.
Google-HealthCXR [3] is manually labeled by medical experts for CXR images with high accuracy and contains about 4000 images.
We follow the traditional UDA setting, and choose the disease closed set in these five datasets as multi classification labels, i.e., Atelectasis, Cardiomegaly, Effusion, Consolidation,
Edema and Pneumonia.
Four transfer scenarios are constructed, which are NIH-CXR14, CheXpert,
MIMIC-CXR to Open-i ; NIH-CXR14, CheXpert, MIMIC-CXR to Google-Health-CXR ; CheXpert,
MIMIC-CXR to NIH-CXR14 and NIH-CXR14, CheXpert to Open-i .
Implementation details.
In order to make a compromise between images in different datasets, we scale the images to 128*128 before feeding them into the network.
To expand the training set, several data augmentation techniques are used, including random cropping and horizontal flipping.
SGD with momentum of 0.9 is used as the optimizer.
For the student model, the initial learning rate is 0.01 and the weight decay is 5e-4.
The learning rate for coordinating weight mapping is 0.001;
For the teacher models, the initial learning rate is 0.001 and the weight decay is 5e-6.
The values of α , β and γ are set as 0.5, 0.01 and 0.01 respectively.
For the case when the target domains datasets are small-scale, such as Open-i and Google-Health-CXR , we assume that there are 200 labeled data in the target domains, and in order to give a good initial condition for training, we randomly select a source model to initialize the target model.
For the case when the target domains datasets are large-scale, such as NIH-CXR14 , we assume that there are 500 labeled data in the target domains.
Unless otherwise specified, the interval for updating coordinating weight mapping is set as 100 iterations.
Following the setting of multi-label medical image classification problems, the evaluation criterion is Area
Under the Receiver Operating Characteristic (AUROC) [15] curve score. [Table] [Table]
4.1 Comparisons to State-of-the-Art
At present [Table] there does not exist any experimental report on our problem setting.
So we choose four category of methods for compare.
The first category is Source only which means directly applying a teacher model to the target domain.
The second category is Fine-tune( average ) which fine-tune each teacher network using labeled target domain data, then average their predicted values.
The third category is the state-of-the-art multi-source-free domain adaptation methods, which are
DECISION [1], CAiDA [14], and SHOT-best.
The SHOT-best refers to adapting each source domain separately through the SHOT [35] method.
The model with the best performance on the validation set is selected.
The final category is semi-supervised domain adaptation methods, which are MME [50] and ECACL [30].
For the semi-supervised domain adaptation methods, we assume that the labeled target data are the same as our method.
Since they are single-source based methods, we perform domain adaptation for each source model and take the best result.
Tables 1-4 show the comparison results on four transfer scenarios.
MetaTeacher(all) is our proposed method.
Source Only(N), Source Only(C) and Source Only(M) are the teacher models respect to the NIH-CXR14 , CheXpert and MIMIC-CXR datasets respectively. For the scenario from CheXpert ,
MIMIC-CXR to NIH-CXR14 , since the dataset NIH-CXR14 contains 108,948 x-ray images, different from other scenarios, this time we do not need to initialize the target model with the source models.
It can be observed that our method achieves the best performance.
The extensive experiments on four different transfer scenarios verify the adaptability of our method under multi-label chest x-ray dataset transfer cases.
For the scenario from NIH-CXR14 , CheXpert to Open-i , as show in Table 4, the
[Table] [Table]
performance of two source domains is 0.86% lower than that of three source domains.
Furthermore,
MetaTeacher also has moderate training time and more clearer background (see Appendix) [Table]
4.2 Ablation Analysis and Discussion
Component analysis.
In Tables 1-4, MetaTeacher(w/o mapping ) represents that our proposed method removes the part of coordinating weight learning and optimization substituted by average.
MetaTeacher(w/o update ) means to remove the bilevel optimization process.
In this situation, the weighted output of teachers is used to supervise the learning of student network.
The results in the last three rows of Tables 1-4 show that these two parts are indispensable.
It is worth mentioning that MetaTeacher(w/o mapping ) still obtains promising performance due to the following reasons.
First, for student updating, averaging predictions from multiple teachers is beneficial for student performance, consistent with the finding by [68].
Second, the fixed W is also involved in the teacher optimization.
It means bilevel optimization contributes more gain to the overall performance than the coordinating weight learning.
However, the coordinating weight learning can judge which disease category the teacher is good at by weight, knowledge with different weights can be learned from different teachers.
Therefore, the results in each disease category are close to the predictions of the best teachers, such as Pneumonia in Table 1 and Atelectasis in Table 2 (also see Appendix). Effects of proportion of labeled target data.
Table 5 shows the influence of the amount of labeled data in the target domain on the transfer scenario of NIH-CXR14, CheXpert, MIMIC-CXR to Open-i .
The experimental results show that the performance slowly improves as the amount of labeled data increases; a small number of labeled target domain samples can achieve good results.
[Table]
[Figure] [Table] [Figure]
[Table] [Figure] [Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
Parameter analysis.
We conduct parameter analysis experiment on the transfer scenario of NIHCXR14, CheXpert, MIMIC-CXR to Open-i .
The basic strategy is to change a parameter while other parameters are fixed.
Our method MetaTeacher has three hyperparameters, i.e., α and β in Eq.
(3), and γ in Eq.(11).
Fig.(2)(a) shows performance changing with the parameter α .
When α = 0 , the coordinating weight mapping is not trained effectively resulting in the inability to determine the optimization direction of each teacher.
When α gradually increases to around 0.5, the result achieve optimal performance.
Fig.(2)(b) shows the influences of the parameters β .
When the β is too large, it means that the coordinating weight learning part is ineffective and cannot express the relationship between the source domains.
When β is set to 0, coordinating weight learning may overfit, which may cause coordinating weights to work well on some instances but poorly on other instances; for this case, the performance is 92.49% about 0.35% lower than the result 92.84% in Table 1. Fig.(2)(c) shows the influences of the parameter γ on divergence loss.
When γ is set to 0.01, the performance reaches the best, but with the continuous increase of γ , the performance decreases obviously.
When γ = 0 , the result is 92.29%, which is 0.55% lower.
We can also see that our method is also quite stable for the parameters α, β and γ in a large interval.
Conclusion
In this paper, we have proposed a novel framework, termed as MetaTeacher, for semi-supervised multi-source-free domain adaptation for medical image classification.
The transfer learning process is modeled as a multi-teacher and one-student scheme.
We not only optimize the student, but also optimize the teachers through the student’s feedback in the target domain.
Our optimization is based on meta-learning with two main parts: coordinating weight learning, and bilevel optimization.
The first part obtains the coordinating weight mapping which is then used to coordinate the teacher outputs and updates.
Bilevel optimization updates the student based on the pseudo-labeled data produced by the teachers and updates each teacher based on the feedback signal generated by the student and other teachers.
Extensive experiments on multi-label chest x-ray datasets empirically demonstrated the superiority of our method over many state-of-the-art approaches. Acknowledgments and Disclosure of Funding
This work was supported in part by the National Key R&D Program of China (2018YFE0203900),
National Natural Science Foundation of China (62276048), Sichuan Science and Technology Program (2020YFG0476).
Appendix A Updating Rules of Bilevel Optimization
We follow the derivation route in [47] except the coordinating weight part.
For expression clarity, let | T i | and | S | denote the dimensions of θ T i and θ S respectively, where θ S ∈ R | S |× 1 and θ T i ∈ R | T i |× 1 . Suppose there is a batch of unlabeled target samples x tu , the j -th teacher T j samples the pseudo labels ˆ y ju ∼ f T i ( x tu ; θ T j ) for j = 1 , · · · , n and [Equation] (12) where W ju is the j -th row of coordinating weight matrix W u w.r.t x tu and ◦ is the Hadamard product.
So we can use ( x tu , ¯ y tu ) to update the parameter θ S according to Eq.(6) in expectation as follows,
[Equation]
[Equation]
According to Eq.(7), we update θ T i based on a batch of labeled target data ( x tl , y l ) by optimization the following objective function
[Equation]
For end-to-end optimization with gradient descent, the partial derivative respect to the above objective function R is
[Equation] for i = 1 , · · · n . According to the chain rule, Eq.(15) can be written as: [Equation]
[Equation] For the right part of Eq.(16) [Equation] it follows that
[Equation]
where ( · ) T is the transpose notation.
Suppose that
[Equation]
it follows that
[Equation]
Since ¯
[Equation]
W ju
[Equation]
Eq
[Equation]
(19) can be further resolved
[Equation]
From the REINFORCE equation [60]
[Equation]
we can get
[Equation]
where ˜ y iu is the pseudo labels after normalizing the values of f T i ( x tu ; θ T i ) ·
W iu to 0 or 1, i.e., ˜ y iu,j =when ˆ y iu,j < 0 .
5 and ˜ y
iu,j = 1 for other cases.
After substituting Eq.(21) into Eq.(19), Eq.
(18) into
Eq.(19), and Eq.(19) into Eq.(16)
, we obtain the following gradient,
[Equation]
By Monte Carlo approximation, we use the sampled ˆ y ju for j = 1 , · · · , n to obtain the update rules,
[Equation]
B Visualization B.1 Visualization of ablation analysis
We visualize the domain adaptation performance on the transfer scenario NIH-CXR14, CheXpert,
MIMIC-CXR to Open-i .
The visualization sample in the Open-i is suffering from Atelecsis and
Effusion disease.
The comparison models are 1) Source only(N): the model trained on the NIHCXR14 dataset; 2) Source only(C): the model trained on the Chexpert dataset; 3) Source only(M): the model trained on MIMIC-CXR dataset; 4) MetaTeacher( w/o update ): our approach only containing coordinating weight learning part; 5) MetaTeacher( all ): our full approach MetaTeacher containing both coordinating weight learning and bilevel optimization.
Atelectasis
Cardiomegaly
Effusion
Consolidation
Edame
Pneumonia
M e t a T eac h e r( a ll )
M e t a T eac h e r( w / o u pd a t e )
S ou r ce
O n l y ( M )
S ou r ce
O n l y ( C )
S ou r ce
O n l y ( N )
From the visualization results, we have the following observations.
The source models trained on different datasets have different concerns about different diseases.
It can be seen that Source only(N) and Source only(M) can identify patients with Effusion disease, with probabilities of 76.513741% and 62.620537%, respectively. However, Source only(C) shows that the patient has only a 48.198967% probability of Effusion disease. 2) Simply fusing multiple teacher predictions does not work in the target domain. MetaTeacher( w/o update ) is a distillation learning with coordinating weights, which can coordinate the knowledge of each teacher about each disease category.
As shown in Fig.3, if most of the source models can detect some disease, the fused model can also detect this disease, and its probability is slightly lower than the maximum value such as Effusion disease.
Conversely, if the disease cannot be detected by most of the source models, the fused model can not detected it too such as Atelectasis, Consolidation and Edame. 3)
Collaboratively updating teacher and student models works in the target domain.
MetaTeacher( all ) can learn knowledge that the source model does not have.
None of these three source domain models can accurately detect the Atelectasis disease, but
MetaTeacher( all ) can identify it, and the output probability is as high as 79.616920%.
In addition, for
Consolidation, Edame and Pneumonia diseases, MetaTeacher( all ) predictions for them are close to 0, which shows that our method has a more clear judgment ability for non-existing diseases.
B.2 Visualization of different methods
The comparison models are DECISION [1] and MME [50].
The first method is a multi-source-free domain adaptation approach, which works by learning a set of weight values corresponding to each
M e t a T eac h e r D E C I S I
ON MM E (a) (b) MM E D E C
I S I ON
M e t a T eac h e r Atelectasis Cardiomegaly Effusion
Consolidation
Edema Pneumonia Atelectasis Cardiomegaly
Effusion
Consolidation Edema Pneumonia source domain model, while learning these weights by using unlabeled target data, then combining the predictions from each source domain as the final prediction. To fit the problem setting, the performance of DECISION is visualized on the transfer scenario NIH-CXR14, CheXpert,MIMICCXR to Open-i .
The second method is a single-source semi-supervised domain adaptation approach, which alternately maximizing the conditional entropy of unlabeled target data with respect to the classifier and minimizing it with respect to the feature encoder.
Similarly to fit the problem setting, we visualize the MME performance on the transfer scenario MIMIC-CXR to Open-i .
The visualization sample in Fig.4(a) is suffering from Atelecsis and Effusion disease while the sample in Fig.4(b) is suffering from Cardiomegaly disease.
[Table]
As shown in Fig.4, both of MME and DECISION cannot detect the corresponding diseases.
From the visualization results, it can be seen that MME and DECISION contain a widely distributed yellow color, mixed with the red part, which affects their judgments of the disease. For example, Atelectasis and Effusion diseases in Fig.4(a), or Cardiomegaly disease in Fig.4(b), although MME and DECISION can mark the disease location in red, they also contain a lot of yellow color in other places, which confuse their attentions to the right diseases. Unlike them, MetaTeacher contains more blue background color, which can more clearly distinguish the background color from the disease location.
Therefore, the disease can be judged more accurately.
Additionally, for diseases that are clearly not present in the figure, such as Consolidation, Edema and Pneumonia diseases in Fig.4(a), or Atelectasis and Pneumonia in Fig.4(b).
The widespread yellow color makes MME and DECISION more conservative in their predicted probabilities, while the predicted probabilities by MetaTeacher are closer to 0 compared to them.
From the visualization results, it can be seen that MetaTeacher is more accurate. C More Discussions C.1 Training Time Comparison The training runtime of MetaTeacher is compared with DECISION [1] and MME [50] on a single NVIDIA 3090Ti GPU over the transfer scenarios NIH-CXR14, CheXpert, MIMIC-CXR to Open-i and
NIH-CXR14, CheXpert, MIMIC-CXR to Google-Health-CXR . The results are shown in Table 6. It can be observed that our method MetaTeacher is slightly slower than the approach of multi-source-free domain adaptation (e.g., DECISION). Although multi-source-free domain adaptation methods do not need to update a student model, they involve other complex designs (e.g., DECISION need to do k-means clustering, and CAiDA [14] has a searching process of Semantic-Nearest Confident Anchor).
Instead, MetaTeacher only involves some simple matrix calculation.
The second observation is that
MetaTeacher is slightly faster than the approach of semi-supervised domain adaptation (MME).
This is because semi-supervised domain adaptation needs to train a model for each source domain, whilst other complex computations are involved in their optimization process.
Overall, our method has similar running speed as existing alternative methods. C.2 Probing the Behavior of Coordinating Weight The coordinating weight is critical in our MetaTeacher framework. Firstly, for the upper-level optimization objective, it combines the predictions of multiple teachers to provide the updating direction for the student model.
Secondly, for the lower-level optimization objective, we split coordinating weight into multiple vectors to provide different updating directions for each teacher.
On the training of the transfer scenario NIH-CXR14, CheXpert, MIMIC-CXR to Open-i , we choose a sample labeled with Atelectasis and Effusion disease classes to inspect the behavior of coordinating weight.
As shown in Table 7, initially, each teacher, as well as their joint predictions ( ¯ y tu in Eq.
(12)), failed to predict the Atelectasis disease.
During training, each teacher was updated, with teacher 1 acquiring the ability to predict Atelectasis disease (0.371739 to 0.756016).
Meanwhile, the coordinating weight was also accordingly updated and assigned more weight to the Atelectasis class for teacher 1 (0.214039 to 0.950953).
This process is summarized in Table 7.
C.3 Performance Comparisons from Two Sources to Three Sources
Compared with multi-source-free domain adaptation methods, our MetaTeacher yields more significant gains in the multi-source transfer situation.
On the two-teacher (Table 4) and three-teacher (Table 1) scenario, DECISION [1] method increases from 91.26% to 91.67% (a gain of 0.41%), CAiDA [14] method from 90.80% to 90.99% (0.19%), vs. our MetaTeacher from
[Table]
98% to
[Table] [Table] This clearly shows that our performance gain is more significant than those by prior art methods. [Table]
[Table]
[Table] For further validation [Table] we have added a two-teacher transfer experiment: CheXpert, MIMIC-CXR to
Google-Health-CXR . From the two-teacher case to the three-teacher transfer scenario ( NIH-CXR14, CheXpert, MIMIC-CXR to it Google-Health-CXR) [Table]
the performance gain of DECISION is 0.69%, vs.
1.35% by our MetaTeacher (see Table 8). This suggests that our method is superior at leveraging the diversity and complementary effect of multiple teacher models [Table] C.4 Comparison with One-Teacher and One-Student Framework
Assuming no data privacy issue as discussed above
[Table] we experiment with a one-teacher one-student design where the component of adaptively training the teachers goes away naturally.
We obtain the results of 89.97%/79.94%/75.38%/90.13%, inferior to 92.84%/82.69%/77.74%/91.98% by our
MetaTeacher (corresponding to Tables 1,2,3,4 in the main paper). This is due to that each dataset presents unique characteristics in category imbalance and labeling error (e
[Table]
g., false negatives), resulting in different per-category qualities.
Aggregating such datasets into one would introduce negative interference.
Besides, using multiple teachers would reduce the learning difficulties of the entire classification problem in a spirit of divide-and-conquer principle, in addition to the opportunity of modeling the confidence per teacher.
It should be noted that our multi-teacher setup is underpinned by the nature of our problem where data privacy protection is fundamentally critical (i.e., data sharing across hospitals is typically banned).
That being said, a specific teacher model would be trained using the training data of each individual hospital. This leads to the result of multiple teacher models in practice.
Compared to the existing semi-supervised methods, our method requires less labeled data.
For example, in the transfer scenario CheXpert, MIMIC- CXR to NIH-CXR14 , for achieving 77.74% classification rate, the existing semi-supervised methods [2,36,37,39] require about 20,000 labeled samples (20% of the total), vs. our method only needing 500 labeled samples.
Hence, our MetaTeacher is more data efficient and favored in practice.