Matching Distributions between Model and Data: Cross-domain Knowledge Distillation for Unsupervised Domain Adaptation (2021)

Abstract: Unsupervised Domain Adaptation (UDA) aims to transfer the knowledge of source domain to the unlabeled target domain. Existing methods typically require to learn to adapt the target model by exploiting the source data and sharing the network architecture across domains. However, this pipeline makes the source data risky and is inflexible for deploying the target model. This paper tackles a novel setting where only a trained source model is available and different network architectures can be adapted for target domain in terms of deployment environments. We propose a generic framework named Cross-domain Knowledge Distillation (CdKD) without needing any source data. CdKD matches the joint distributions between a trained source model and a set of target data during distilling the knowledge from the source model to the target domain. As a type of important knowledge in the source domain, for the first time, the gradient information is exploited to boost the transfer performance. Experiments on cross-domain text classification demonstrate that CdKD achieves superior performance, which verifies the effectiveness in this novel setting.

## introduction annotating sufficient training data is usually an expensive and time-consuming work for diverse application domains. unsupervised domain adaptation (uda) aims at solving this learning problem in the unlabeled target domain by utilizing the abundant knowledge in an existing domain called source domain, even when these domains may have different distributions. this technique has motivated research on cross-domain text classification @xcite @xcite . one of the important knowledge in the source domain is the labels of samples. current methods mainly leverage the labeled source * corresponding author. unfortunately, sometimes we are forbidden access to the source data, which are distributed on different devices and usually contain private information, e.g., user profile. existing methods cannot solve the uda problem without the source data yet. in addition, it is necessary to adapt the target domain with a flexible network architecture different from the source domain in terms of different deployment requirements for different domains. but most of works @xcite are required to share the same network architecture between different domains. in this paper, we propose a novel uda setting: only a trained source model and a set of unlabeled target data are provided, and the target model is allowed to have different network architectures with the trained source model. it differs from the vanilla uda in that a trained source model instead of source data is provided as supervision to the unlabeled target domain when learning to adapt the model. such a setting satisfies privacy policy and effective delivery, and helps deploy the target model flexibly according to the target application. our setting seems somewhat similar to knowledge distillation (kd) @xcite , where a trained teacher model teaches a student model with different architecture on the same task over a set of unlabeled data. kd assumes that the empirical distribution of the data used for training the student model matches the distribution associated with the trained teacher model. nevertheless, in our setting, the unlabeled data and teacher (source) model have different distributions. one of simple yet generic solution for our setting is to match the distributions between source and target domains under the process of distilling the knowledge. how-ever, it is quite challenging to reduce the shifts between a known distribution (e.g., a trained source model) and the empirical distribution of data (e.g., target data). prior methods minimize a distance metric of domain discrepancy, such as maximum mean discrepancy (mmd) @xcite to match the distributions across domains in terms of the source and target data. unfortunately, the empirical evaluation of these metrics is unavailable since we cannot access the source data. in this paper, we propose a generic framework named cross-domain knowledge distillation (cdkd). specifically, we define a joint kernelized stein discrepancy (jksd) that measures the largest discrepancy over the hilbert space of functions between empirical sample expectations of target domain and source distribution expectations. inspired by the works @xcite , the source distribution expectations are being zero via the effect of stein operator such that we can evaluate the discrepancy of joint distributions without any source data. we embed jksd criterion into deep network where multi-view features including activations, gradients and class probabilities in the source model are exploited to explore the domain-invariant and discriminative features across domains. in addition, we further maximize jksd using adversarial strategy where the multi-view features are integrated into domain adaptation abundantly. finally, cdkd is learnt by joint optimizing both kd objective @xcite and jksd. the main contributions are outlined as, ## methodology we address the unsupervised domain adaptation (uda) task with only a trained source model and without access to source data. we consider k-way classification. formally, in this novel setting, we are given a trained source model f s : @xmath0. here, the goal of cross-domain knowledge distillation (cdkd) is to learn a target model f t : @xmath1, with only d t and f s available. the target model f t is allowed to have different network architecture with f s . cdkd is a special kd which consists of a trained teacher model f s , a student model f t and unlabeled data d t as well. but it differs from kd in that the empirical distribution of d t don't match the distribution associated with the trained model f s . therefore, it is necessary to introduce distribution adaptation to eliminate the biases between the source and target domains during distilling the knowledge. specifically, as shown in figure 1(a) , we first introduce kd to distill the knowledge to the target domain in terms of the class probabilities produced by the source model f s . then, we introduce a novel criterion jksd to match the joint distributions across domains by evaluating the shift between a known distribution and a set of data. this is the first work to explore the distribution discrepancy between a model and a set of data in uda task. ## conclusion in this paper, we shed a new light on the challenges of uda without needing source data. specifically, we provided a generic framework named cdkd to learn a classification model over a set of unlabeled target data by making use of the knowledge of the activation and gradient information in the trained source model. cdkd learned the collective knowledge across different domains including domain-invariant and discriminative features by matching the joint distributions between a trained source model and a set of target data. experiments for cross-domain text classification testified that cdkd still achieves advantages for uda task though without any source data and improves the performance of kd task when the trained teacher model doesn't match the training data. based on this definition, the inner product ∇ x φ(x), ∇ x φ(x ) f d can be evaluated as similar to @xcite , we can compute h(x, y, x , y ) = λ xy , λ x y f d as, thus, jksd s 2 (p, q) is the expectation of h(x, y, x , y )l(y, y ) over the distribution q, given a set of samples d @xmath2, y i )} m @xmath3, y), we can evaluate s 2 (p, q) as 1 m 2