U NSUPERVISED D OMAIN A DAPTATION WITH I MPUTATION
Anonymous authors
Paper under double-blind review
A BSTRACT
Motivated by practical applications, we consider unsupervised domain adaptation for classiﬁcation problems, in the presence of missing data in the target domain.
More precisely, we focus on the case where there is a domain shift between source and target domains, while some components in the target data are systematically absent.
We propose a way to impute non-stochastic missing data for a classiﬁcation task by leveraging supervision from a complete source domain through domain adaptation.
We introduce a single model performing joint domain adaptation, imputation and classiﬁcation which is shown to perform well under various representative divergence families ( H -divergence, Optimal Transport).
We perform experiments on two families of datasets: a classical digit classiﬁcation benchmark commonly used in domain adaptation papers and real world digital advertising datasets, on which we evaluate our model’s classiﬁcation performance in an unsupervised setting.
We analyze its behavior showing the beneﬁt of explicitly imputing non-stochastic missing data jointly with domain adaptation.
I NTRODUCTION
When dealing with machine learning applications in the real world, data usually come with several imperfections that make classical algorithms hardly deployable.
One of these issues is that data are often incomplete.
Typically, while capturing data coming from different locations with several sensors per location, a sensor may randomly fail or even may be just missing at a given location.
Such a situation can also occur in disease diagnosis in multi-modal medical imaging where one of the modalities fails or is not available; for example the positron emission tomography (PET) modality which reveals metabolic information for clinical tests requires ingesting a radioactive tracer which poses health risks and is often missing (Cai et al., 2018).
Similarly, in computational advertising applications, information is missing for users who do not have a prior history on a merchant’s website while their global clicking behavior across websites may be known.
Another common issue is that data used for training and deployment may differ in their generation process: data may be collected on different devices, background noise or compression schemes may affect differently training or deployment data, leading to a shift in data distribution.
This has given rise to the important literature of Domain Adaptation (Pan & Yang, 2010; Kouw & Loog, 2019).
These two issues are usually independently addressed by developing models handling only the missing data or the domain adaptation problem.
In this paper, motivated by practical advertising applications, we consider unsupervised domain adaptation (i.e. labels are not available in the target domain) for classiﬁcation when (1) part of input data is missing in the target domain thus requiring some form of imputation, (2) there is no possible supervision in the target domain for imputation thus requiring indirect supervision from the source domain, and (3) there exists a domain shift between the source and target distributions requiring domain adaptation.
More precisely we consider this adaptation-imputation setting for nonstochastic missing data, i.e. when the same features are missing for all target samples.
This contrasts with many imputation problems which take beneﬁt of stochasticity in missing features.
We propose a model that handles unsupervised domain shift and missing data assuming nonstochastic missing data in the target domain.
The model learns to perform imputation for the target domain while aligning the distributions of the source and target domains in a latent space, thus going beyond the simple juxtaposition of a data imputation module followed by a domain-invariant feature representation learning module.
Imputation makes use of an indirect supervision from the complete source domain.
This key property allows us to handle non-stochastic missing data, while satisfying the constraints related to adaptation and to the classiﬁcation objective.
The imputation process plays several roles in our global architecture as it provides us with information about the missing data for the target domain while contributing to the domain-invariant loss and the reconstruction loss.
Extensive empirical evidence on handwritten digits and Click-Through-Rate prediction (CTR) domain adaptation problems illustrate the beneﬁt of the proposed model.
The original contributions are the following:
• We introduce a new problem : joint unsupervised domain adaptation and imputation for classiﬁcation motivated by practical applications; • We propose a new model for handling the problem end-to-end.
It learns to generate relevant missing information while aligning source and target distributions in a latent space and to classify source instances; • We evaluate the model not only on academic benchmarks but also on challenging real world advertising data.
R ELATED WORK
We review below typical related work for domain adaptation and data imputation. 2. U NSUPERVISED D OMAIN A DAPTATION (UDA) A number of shallow learning methods approach Domain Adaptation by weighting individual observations during training.
These methods focus either on data importance-weighting (Cortes & Mohri, 2014; Zadrozny, 2014) or on class importance-weighting (Z. Lipton & Smola, 2018).
Recent deep learning methods try to align the distributions of the two domains, for example by embedding them in a joint latent space.
There are two main directions for learning joint embeddings.
One is based on adversarial training, making use of GAN extensions.
The other one directly exploits explicit distance measures between distributions such as Wasserstein or Maximum Mean Discrepancy (MMD).
For the former, the seminal work of Ganin & Lempitsky (2015) learns to map source and target domains onto a common embedding space, by optimizing a double objective: on the one hand they minimize an approximation of the H -divergence between the source and target embeddings via adversarial training, on the other hand they learn to classify the source data embeddings.
This inﬂuential work has been followed by several extensions and variants.
ADDA (Tzeng et al., 2017) advocates the use of two different mappings for the source and the target domains based on the argument that this is more suitable when the marginals are different in the two domains.
Liu & Tuzel (2016) trains coupled generative adversarial network (CoGAN) for learning a joint distribution of multi-domain images, that can be used for UDA.
Bousmalis et al.
(2017) use a generator to map the source to the target domain while training the classiﬁer on the learned representations using source labels.
CDAN (Long et al., 2017) improves the domain discriminator by conditioning it on classiﬁer predictions.
A second family of approaches proposes metric based divergences such as MMD (Long et al., 2015) for measuring the loss between source and target representations.
DeepJDOT (Damodaran & Kellenberger, 2018) makes use of an optimal transport formulation to align the joint distributions in a latent space.
In addition to feature alignment they perform label distribution alignment following
Courty et al.
All these works rely on the assumption of covariate shift and consider that full input data is available for both source and target domains.
Our two models ( ADV and OT ) can be seen respectively as extensions of Ganin & Lempitsky (2015) and Damodaran & Kellenberger (2018) for the missing data problem.
2. I MPUTATION
Data imputation is a classical problem addressed by several methods (Little & Rubin, 2002; Van Buuren, 2018; Murray, 2018).
The usual setting is different from ours since it considers reconstructing the whole missing data in the input space, while we consider 1) reconstruction in a latent space and 2) partial reconstruction since we are interested in the information relevant to the classiﬁcation task only.
Recent generative models like GANs (Goodfellow et al., 2014) or VAEs (Kingma & Welling, 2013; Rezende et al., 2014) have been adapted for data imputation in Yoon et al.
and Mattei &
Frellsen (2019) respectively.
GAIN (Yoon et al., 2018) is an extension of conditional GANs where the generator takes as input an incomplete data and performs imputation while the discriminator is trained to guess for each sample if each variable is original or imputed.
Mattei & Frellsen (2019) suggests a method based on deep latent variable models and importance sampling that offers tighter likelihood bound compared to the standard VAE bound.
Most approaches consider a supervised setting where 1) paired complete and incomplete data are available and 2) missingness corresponds to a stochastic process (e.g. a mask distribution for tabular data), 3) imputation is performed in the original feature space.
Note that this is different from our setting where there is no direct supervision (supervision is only provided indirectly through the source domain) and missingness is non-stochastic which makes the problem harder since one cannot compute statistics on different incomplete samples.
The general approach with generative models is to learn a distribution over imputed data which is similar to the one of plain data.
This comes in many different instances and usually, generative training alone is not sufﬁcient; additional loss terms are often used.
In paired problems, i.e. when each missing datum is associated to a plain version of the datum, these additional terms consist of a reconstruction term imposed by a MSE contraint (Isola et al., 2016b).
In unpaired problems a cycle-consistency loss is imposed as in Zhu et al.
Li et al.
; Pajot et al.
(2019) are among the very few approaches addressing unsupervised imputation in which full instances are never directly used.
Both extend the AmbientGAN (Bora et al., 2018) framework and consider stochastic missingness.
Our imputation problem is closer to the ones addressed in some forms of inpainting or for multimodality missing data.
The former problem is addressed e.g. in Pathak et al.
(2016) who proposes an encoder-decoder model trained according to a joint reconstruction and adversarial loss.
The latter is addressed in Cai et al.
(2018) who considers the case of multi-modality when one or more modalities are systematically absent, but they do not consider adaptation.
They propose to learn to reconstruct the missing modality distribution conditionally to the observed one.
Both approaches are fully supervised.
Ding et al.
(2014) is the only paper we are aware of that considers imputation as we do.
Their approach is based on low rank constraints and dictionary learning to guide the transfer between domains.
We do not use this method as a baseline due to the complexity and running time of this method which relies on singular value decompositions and dictionnary learning.
P ROBLEM DEFINITION
Let us denote respectively ( x S , y S )
∈ R n × R and ( x T , y T ) ∈
R n × R , data from the source and target domains where x − is an input, y − the associated label and n is the dimension of the input space.
“ − ” holds for either source S or target T .
The joint distribution on each domain is denoted respectively p S ( X, Y ) and p T ( X, Y ) .
We consider that x − has two components, x − =
( x − 1 , x − 2 ) .
The problem we address is Unsupervised Domain Adaptation (UDA) with missing features in the target domain.
More precisely, we make the following hypotheses.
Missingness: We assume that features are the same across domains and that source features x S
=
( x S 1 , x S 2 ) are always available while in the target domain only x T 1 is available and x T 2 is systematically missing.
For advertising applications for example, x would characterize the user browsing behaviour on merchant sites; x −characterizes global user features aggregated over his navigation history, which are known for all users; x − 2 characterizes user history on a target merchant site.
Source domain would consist of all users who already visited this merchant site and target domain of users who never visited this site.
UDA: we assume that source labels y S are available whereas target labels y T are unknown.
Covariate shift: we assume covariate shift as in most UDA papers e.g. Ganin & Lempitsky (2015).
A DAPTATION -I MPUTATION MODEL
As in many generative approaches to UDA, the objective is to project source and target data onto a common latent space in which data distributions from the two domains match, and to learn a classiﬁer on the source data that performs well on the target domain.
The novelty of our approach is to offer a solution to deal with datasets in which some information, x T 2 , is systematically missing in the target domain.
Our model, denoted Adaptation-Imputation , performs three operations jointly: imputation of missing information for the target data, alignment of the distributions of source and target, and classiﬁcation of source instances.
The three operations are performed in a joint embedding space and all the model’s components are trained together.
The term imputation is used here in a broad sense: our goal is not to recover the whole missing x T 2 , but to recover information from x T 2 that will be useful for adaptation and for the target data classiﬁcation objective.
This is achieved via a generative model, which for a given datum in the target domain and conditionally on the available information x T 1 , attempts to generate the required missing information.
Because x T 2 is systematically missing for target data, there is no possible supervision in the target domain; instead we use distant supervision from the source data while transferring to the target domain.
We consider two variants of the same model based on different divergence measures between source and target distributions: the Wasserstein distance and the H -divergence approximated through adversarial training.
For simplicity we describe in the main text the adversarial version ADV and defer the
Optimal Transport OT description to Appendix B.
We report the results obtained with both models in Section 5.
4. T RAINING
Our model is composed of three different modules responsible for adaptation, imputation and classiﬁcation, that share parameters and are trained in parallel.
For simplicity, we describe each component in turn, but it should be reminded that they all interact and that their parameters are all optimized according to the three objectives mentioned above.
The interaction is discussed after the individual module descriptions.
The model’s components are illustrated in Figure 1 (a).
Adaptation The latent space representations of source and target domains are denoted with a tilde notation: (cid:101) x S =
( (cid:101) x S 1 , (cid:101) x S 2 ) and (cid:101) x T =
( (cid:101) x T 1 , (cid:101) x T 2 ) .
Referring to Figure 1 (a), (cid:101) x − 1 =
g 1 ( x − 1 ) , ( (cid:101) x − 1 denotes either (cid:101) x S 1 or (cid:101) x T 1 ) is the mapping of the observed component x − 1 onto the latent space and (cid:101) x −
=
h ◦ g 1 ( x − 1 ) is the second component’s latent representation generated from x − 1 .
This generation mechanism will be described later.
Adaptation aligns the distributions ( (cid:101) x S 1 , (cid:101) x S 2 ) and ( (cid:101) x T 1 , (cid:101) x T 2 ) in the latent space.
For the ADV model, alignment is performed via a classical adversarial loss operating on the latent representations:
[Equation]
where D 1 ( (cid:101) x ) represents the probability that (cid:101) x comes from the source rather than the target.
Imputation Imputation amounts at generating an encoding (cid:101) x
T 2 , in the latent space, for the missing information in the target data, conditioned on the available information x T 1 .
Our objective here is to generate missing information which is relevant for the classiﬁcation objective.
Since we never have access to any target component x T 2 , we learn to perform imputation based on the source data.
More precisely, we learn to generate (cid:101)
x S 2 from x S 1 through the generator h , (cid:101) x S
= h ◦ g 1 ( x S 1 ) , as depicted in Figure 1.
We want h to generate the missing information (cid:101) x S 2 associated to the observed x S 1 .
For that we perform two operations in parallel.
First, we align the distribution of (cid:101) x
S 2 with the distribution of (cid:98) x S 2 = g 2 ( x S 2 ) , that is a direct mapping of x S 2 onto the shared latent space, using an adversarial loss described below.
The intuition is that both g 1 and g 2 are simple mappings operating respectively on x S 1 and x S 2 while h acts as a generator conditioned on x S 1 for generating (cid:101)
x S 2 .
Moreover, we not only impose this distribution alignment, but would also like (cid:101)
x
S 2 to represent missing information relative to x S 2 and associated to a speciﬁc x S 1 .
For that, we use a reconstruction term in parallel to the above alignment, in our case a MSE distance between (cid:101) x S 2 and (cid:98) x S 2 .
This MSE term guarantees that the imputed (cid:101) x S 2 truly represents information present in x S 2 .
Similar ideas combine distribution matching and MSE conditioning and have been used e.g. in Isola et al.
(2016a)
; Pathak et al. (2016).
The learned mappings are used to perform imputation on the target data (cid:101)
x T 2 =
h ◦ g 1 ( x T 1 ) .
The imputation loss has thus two components.
The ﬁrst is the adversarial term L ADV responsible for aligning (cid:101) x
S 2 and (cid:98) x S 2 , L ADV =
E x 2 ∼ p S ( X 2 ) log D 2 ( (cid:98) x S 2 )
+ E x 1 ∼ p S
X 1 ) log(1 − D 2 ( (cid:101) x S 2 )) .
The second is the reconstruction term L MSE =
E x ∼ p S ( X ) (cid:107) (cid:101) x
S 2 − (cid:98) x S 2 (cid:107) 2 2 .
The total imputation loss is then:
[Equation]
where λ ADV , λ MSE are hyperparameters.
The two processes of imputation and adaptation inﬂuence each other.
Both are also inﬂuenced by the classiﬁcation process described below.
Its effect on imputation is to force the generated (cid:101) x
S 2 to contain information about x S 2 relevant for the classiﬁcation task.
This information is transferred via adaptation to the target domain when generating (cid:101) x T 2 .
Classiﬁcation The last component of the model is a classiﬁer f , trained on the source domain mapping (cid:101)
x S for the classiﬁcation task as classically done for UDA.
The corresponding loss is:
[Equation]
where L Disc is typically a cross-entropy loss.
Overall loss The overall loss function L will be the weighted sum of the adaptation, imputation and classiﬁcation losses:
[Equation]
where λ 1 , λ 2 , λ 3 are hyperparameters and the ﬁnal optimization problem is:
[Equation]
Interaction between the model’s components Both mappings g 1 , g 2 and generator h appear in the three terms of the loss function in Equation 4, meaning that they should learn to perform the three tasks simultaneously.
g 1 learns to map the x S 1 and x T 1 components onto the latent space, the mappings being denoted respectively (cid:101)
x S 1 and (cid:101) x T 1 .
h learns to generate missing information (cid:101)
x
T 2 from (cid:101) x T 1 .
The formed (cid:101) x − is generated such that it fulﬁlls the classiﬁcation objective.
g 2 on its side should fulﬁll the imputation objective while preserving part of the information present in x S 2 .
Note that our model makes use of a unique mapping g 1 for both source and target domains.
Separate mappings could have been used for the two domains, but the proposed solution was found to be more robust and to reduce the number of parameters during learning.
[Figure]
inference.
Implementation Let us now detail the implementation of this model.
For adversarial training, discriminators D 1 (adaptation) and D 2 (imputation) will be implemented by binary classiﬁers.
Dis trained to distinguish between source (cid:101) x S and target (cid:101) x T mappings while D 2 is trained to separate imputed (cid:101)
x S 2 , generated from x S 1 , from (cid:98) x S 2 a direct embedding of x S 2 .
We use the gradient reversal trick in Ganin & Lempitsky (2015) for implementing the min-max condition and deﬁne two gradient reversal networks on D 1 and D 2 .
We use an adaptive update of the scale of the gradients in D 1 and
D 2 and optimize L 1 , L 2 and L 3 jointly as synthesized in Algorithm 1 in the Appendix.
In practise we ﬁx all hyperparameters but λ MSE to 1, additional tuning could yield improved performance.
4. I NFERENCE
At inference, given x T 1 , we generate (cid:101) x T
=
( (cid:101) x T 1 , (cid:101) x T 2 ) with (cid:101) x
T 1 = g 1 ( x T 1 ) an embedding of x T 1 and a generated (cid:101) x T 2 , encoding part of the missing information x T 2 in x T , as illustrated in Figure(b).
We use for the latter the following mapping: (cid:101) x
T 2 =
h ◦ g 1 ( x T 1 ) where g 1 is as above and h is the generative mapping conditioned on (cid:101) x T 1 .
Finally (cid:101)
x
T is used as input to the classiﬁer f .
E XPERIMENTS 5. D ATASETS AND EXPERIMENTAL SETTING
Datasets Experiments are performed on two types of datasets.
The ﬁrst one is a classical digits classiﬁcation benchmark used in many domain adaptation studies which we will refer to as digits and transformed to ﬁt our missing data setting.
The second one corresponds to advertising datasets.
The task here is binary classiﬁcation: one wants to predict Click-Through-Rate (CTR) or Conversion
Rate (CR) given user behavior.
This is one of the problem that has initially motivated our adaptationimputation framework.
We use two such datasets: ads-kaggle is a public kaggle dataset 1 , while ads-real has been gathered internally and corresponds to real advertising trafﬁc.
Further details on datasets and preprocessing are presented in Sections 5.2, 5.3 and in Appendix C.
Baselines We report results for the following models:
• Full models: Source-Full is trained without adaptation on the full x S and tested on full x T when the latter is available ( digits ); Adaptation-Full adds adaptation to this model.
• Missing models: Source-Missing and Adaptation-Missing do the same but considering full x S while x T is incomplete: x T =
( x T 1 , 0) , i e. x T 2 is set to 0 .
• Partial models: Source-Partial and Adaptation-Partial is a variant of the above setting where only the ﬁrst component x − 1 for source and target are considered for adaptation and classiﬁcation while the second ones x − 2 are simply ignored.
• Imputation models: Adaptation-Imputation corresponds to our model.
• Naive model: Naive is used for ads-kaggle to provide a reference loss value for this dataset.
It predicts for all examples the mean CTR value as computed using source training data only.
Adaptation-Full is an upper bound of the performance of Adaptation-Imputation since it uses full information while x T 2 is not available in practice.
Adaptation-Missing and Adaptation-Partial can be considered as lower bounds
for
our model since they only
perform adaptation and no imputation
Parameters and architecture of the neural networks used for the different models and experiments are presented in Appendix D. Hyperparameters are chosen using the Deep Embedded Validation estimator introduced in You et al.
combined with heuristics and typical UDA values.
Further details are given in Appendix D.2.1.
We present the results for digits and ads respectively in Sections 5.2, 5.3.
Section 5.4 presents ablation studies.
Reported results are mean value and standard deviation over ﬁve different initializations; best results are indicated in bold .
5. D IGITS
Description For digits , we consider the unsupervised adaptation between two datasets among
MNIST (LeCun et al.
, USPS (Hull (1994)), SVHN (Netzer et al. (2011)) and MNIST-M (Ganin & Lempitsky (2015)).
The direction MNIST → SVHN is not considered as the task is difﬁcult even for traditional UDA (Ganin & Lempitsky, 2015).
All tasks are 10-class classiﬁcation problems.
From the complete image datasets, we build datasets with missing input values.
Half digit missing In a ﬁrst series of experiments, we removed one half of each image, the horizontal bottom part.
We report classiﬁcation accuracy in Table 1 for the different adaptation problems and models ( ADV and OT ).
Removing half of the image leads to a strong performance decrease for Source-Partial and Source-Missing with respect to the upper bounds provided by Source-Full , respectively between 10 and 20 points of accuracy, and between 15 and 45 points.
This is partially recovered when training with adaptation ( Adaptation- Partial ,
Adaptation-Missing , for both ADV or OT ).
But the gap is still important with respect to the upper bound, i.e. Adaptation-Full .
In all cases, Adaptation-Imputation clearly increases the performance; between 10 and 25 points of accuracy over Adaptation-Missing and 2 to 20 points over Adaptation-Partial .
This is a very signiﬁcant improvement which validates the importance of the imputation component.
In Section 5.4 we show that the simultaneous use of imputation and adaptation is required for reaching this level of performance.
Imputation or adaptation alone are well behind the jointly trained instance of the model.
However, it does not reach the upper bound performance of Adaptation-Full where the difference lies between 10 and 25 accuracy points.
Moreorever, Adaptation-Imputation beats the non-adapted Source-Full baseline on several datasets.
Both the ADV and OT versions exhibit the same general behavior.
In the reported results in Table 1, ADV performance is higher than OT .
This is because performance is highly dependent on the NN architectures and we tuned our NNs for ADV .
OT models may reach performance similar to ADV
but we ﬁnd that it requires models with an order of magnitude more parameters.
To keep the comparison fair, we thus used the same NN models for both ADV and OT .
Imputation models achieve their highest performance when the adaptation task between domains is complex (MNIST → MNIST-M, SVHN → MNIST) illustrating the importance of imputation when transfer is difﬁcult.
In all experiments, the performance of --Partial model where ” - ” refers to Source or Adaptation , are usually higher than the --Missing model.
Our understanding is that setting missing components to zero tends to increase distance between source and target distributions, compared to just ignoring them, making the classiﬁcation and adaptation problems harder.
[Table]
Missing patch size In a second group of experiments on digits , we analyze the evolution of the performance of the models with respect to the size of the missing information in the target domain.
For that, we vary the size of the missing patch removing a percentage of the image p with p ∈ { 30% , 40% , 50% , 60% , 70% } on SVHN → MNIST for ADV models, keeping the same hyperparameters as the ones used for p = 50% .
We report the mean values over ﬁve runs in Figure 2.
We notice that Adaptation-Imputation constantly beats the other baselines regardless of the missing patch size.
The ﬁgure exhibits borderline cases when the size of the missing patch be- comes very small ( < 30% ) or very large ( > 65% ).
When the missing patch is too small most of the information for predicting the target label is already available thus simple models perform already well; while when it becomes too big, too few information is available to guarantee efﬁcient reconstructions from the non-missing patch.
5. A DS
Description We performed a second series of tests on two advertising datasets: ads-kaggle and ads-real .
The ads datasets correspond to binary classiﬁcation problems; the task is to predict the probability that a user exposed to an ad from a target partner (e.g. Booking) on a publisher (e.g. NY Times) will click ( ads-kaggle ) or make a purchase ( ads-real ) conditioned on the user history.
A row in the dataset corresponds to a display i.e. an ad opportunity of a click or purchase for a given (user, partner) pair at a given time on a given publisher site.
The source domain is composed of users who already had interactions with a target partner.
The target domain is composed of users with no history on a target partner.
We consider all partners in a given trafﬁc.
For the two domains, x − 1 features correspond to aggregated user features on all the partners, while x − 2 corresponds to user - target partner speciﬁc interaction which is known for the source domain but unknown for the target domain.
Note that besides missingness, there is also an adaptation problem since statistics for new users are usually different from those of known users (e.g. in terms of frequency of a partner’s website visits) as seen in Appendix E.
In real datasets, trafﬁc in the source domain is usually abundant while scarce in the target domain.
Statistics for each dataset are provided in Table 5 in the Appendix; exact preprocessing used is provided in Appendix C.
Results For this group of experiments, we report the results only for ADV models since the trend has been observed to be similar on digits for both ADV and OT .
For ads datasets, missing features do not exist, so we do not report the --Full models’ results on these datasets.
The classes being imbalanced, accuracy is not relevant here so we report another performance measure, cross-entropy (CE) between the predicted values and the true labels on the target domain which is considered as the most reliable metric to estimate revenue.
Note that given the test set size of ads-kaggle , an improvement of 0.001 in logloss is considered as practically signiﬁcant (Wang et al., 2017).
For the ads problem and for large user bases, a small improvement in prediction accuracy can lead to a large increase in a company’s revenue.
For all experiments, we report in Table 2 CE on target test for ads-kaggle and ads-real .
[Table]
A ﬁrst observation is that
Adaptation-Imputation is significantly better than the baselines on both datasets (Table 2).
For ads-kaggle it improves by 2 .
3% the best adaptation model ( Adaptation-Missing ) while for ads-real the improvement reaches 6 .
3% over the best second which happens to be
Source-Partial .
A second observation is that for any model, adaptation consistently improves over the same model without adaptation.
The only exception is the --Partial setting in ads-real .
A third observation is that the missing component indeed contains relevant information: CE performance on source data (not reported in Table 2) shows that Source-Missing which exploits the x − 2 component is consistently higher than Source-Partial which does not exploit this component, leading to relative gains of the former over the latter of 5.6 % on ads-kaggle and 8.2 % on ads-real .
Adaptation-Imputation is able to generate and to exploit this information.
5. A BLATION ANALYSIS
We analyze now the role and importance of the different components of our model, and compare with the results from Tables 1 and 2.
We perform experimentation on the public datasets, digits and ads-kaggle and report results in Table 3 and Figure 3.
Importance of adaptation We compare the performance of the model with and without the adaptation term L 1 in Equation 4.
When removing adaptation, inference is performed as before, by feeding (cid:101)
x T =
( (cid:101) x T 1 , (cid:101) x T 2 ) to the classifer f .
This means that we only rely on the imputation and classiﬁcation losses to learn the parameters of the model.
Results appear on the top of Table 3.
For all datasets, the adaptation component considerably increases the performance, from 10 to 30 points for digits and by a signiﬁcant 0 .
CE value on ads-kaggle .
Imputation mechanism Imputation, cf.
Equation 2, combines adversarial training ( ADV ) and conditioning on the input datum via the MSE loss ( MSE ).
The objective is to learn from x S 1 (cid:101)
x
S 1 = g 1 ( x S 1 ) and to generate missing information in x S 2 , (cid:101) x S
= h ( (cid:101) x S 1 ) .
ADV aligns the distributions of (cid:101) x
S 2 and (cid:98) x S 2 while MSE can be thought as performing some form of regression.
For a given partial information x S 1 , there are possibly several potential x S 2 and thus (cid:101) x
S 2 .
ADV allows to focus on a speciﬁc mode of (cid:98) x
S 2 , while MSE will favour a mean value of the distribution.
Results on Table 3, second group of rows, show that for digits , combining the two inﬂuences ( MSE and ADV ) leads to improved results compared to using separately each loss.
MSE alone already provides good performance, while using only ADV is clearly below.
For this classiﬁcation task, identifying the most relevant mode improves the performance over simple regression ( L MSE ).
Note that reconstruction is an ill posed problem since the task is inherently ambiguous - different digits may be reconstructed from one half of an image.
We performed tests with a stochastic input component in order to recover different modes, but the performance was broadly similar.
Achieving diversity with Conditional
GANs remains an open research topic (Yang et al., 2019).
For the ads-kaggle dataset, the performance of MSE and MSE + ADV are similar.
This is analyzed deeper in an additional series of experiments with several weighted combinations of MSE and ADV .
Results are provided in Table 3 third group of rows, for both digits and ads-kaggle and are plotted for ads-kaggle in Figure 3.
For digits , this conﬁrms that the equal weights selected for our experiments are indeed generally a good choice reducing the burden of hyperparameter selection, while for ads-kaggle performance could be slightly improved with other weightings.
One can see on Figure 3 that ADV induces a high variance in the results (left part of x-axis) while MSE stabilizes the performance (right part of x-axis).
The former allows for better maximum performance but with high variance: performance ranges from 0.35 to 0.7 on the target domain.
A small contribution from MSE (here λ MSE = 0 . 005 )
stabilizes the results.
[Table]
C ONCLUSION
We have proposed a new model to solve unsupervised adaptation problems in the presence of nonstochastic noise in the target domain, by using distant supervision from a complete source domain through domain adaptation and imputing missing values on the target domain in a latent space.
This method uses only labelled source instances and leads to important gains on classical adaptation benchmarks over baseline models for two representative families of divergences (optimal transport, adversarial training).
We have demonstrated on real world advertising datasets that these methods can be used for problems with missing features in advertising.
Potential follow-ups include: extending this method to a semi or fully supervised setting on the target domain; considering simultaneously domain and target shift which frequently occurs in real world problems while still being an open problem; introducing increased diversity in the generation process.
B OT FORMULATION
We present here in more details Adaptation-Imputation using Optimal Transport as a divergence metric.
The formulation is slightly different compared to ADV models.
B. F ORMULATION OF THE IMPUTATION OT MODEL
We replace the H -divergence approximation given by the discriminators D 1 and D 2 by the Wasserstein distance between source and target instances ( D 1 ) and true and imputed feature representations ( D 2 ), following the original ideas in Shen et al.
; Damodaran & Kellenberger (2018).
In practice, we compute the Wasserstein distance using its primal form by ﬁnding a joint coupling matrix γ , using a linear programming approach (Peyr´e et al., 2019).
Adaptation In Damodaran & Kellenberger (2018); Courty et al.
(2017), the optimal transportation problem is formulated on the joint ( X, Y ) distributions.
Similarly to Shen et al.
(2018), in our case, we focus on a plan that acts only on the feature space without taking care of the labels.
This leads to the following optimization problem :
[Equation]
where γ 1 ij is the alignment value between source instance i and target instance j .
Imputation For the imputation part, we keep the reconstruction MSE component in Equationand derive the distribution matching loss as:
[Equation]
where γ 2 ij is the alignment value between source instance i and j .
The ﬁnal imputation loss is:
[Equation]
Classiﬁcation The classiﬁcation term in Equation 3 is unchanged.
B. I MPLEMENTATION
The optimization problem in Equation 5 is solved in two stages following an alternate optimization strategy:
• we ﬁx all parameters but γ 1 and γ 2 and ﬁnd the joint coupling matrices γ 1 and γ 2 using EMD
• we ﬁx γ 1 and γ 2 and solve min g 1 ,g 2 ,h,f L
In practice, we ﬁrst minimize L 3 for a couple of epochs (taken to be 10 for digits ) then minimize
L 1 + L 2 + L 3 in the remaining epochs.
Learning rate and parameters are explained further in Section D.
C D ATASET DESCRIPTION
C.
D IGITS
[Table]
Pre-processing We scale all images to be 32 × 32 and we normalize the input in [ − 1 , 1] .
When adaptation involves a domain with three channels (SVHN or MNIST-M) and a domain with a single channel, we simply triplicate the channel of the latter domain.
As in Damodaran & Kellenberger (2018)
we use balanced source batches which proves to increase performance especially when the source dataset is imbalanced (e.g. SVHN and USPS datasets) while the target dataset (usually
MNIST derived) is balanced.
Scaling the input images enables us to use the same architecture across datasets.
In practise the embedding size is 2048 after preprocessing.
For missing versions, we set pixel values to zero in a given patch.
Figure 4 shows MNIST-M digits with varying missing patch size.
Evaluation The digits datasets are provided with a predeﬁned train / test split.
We report accuracy results on the target test set and use the source test set as validation set (Section D.2.1).
The number of instances in each dataset is reported in Table 4.
We run each model ﬁve times with a different seed for initialization (using Xavier Uniform initialization) and report mean and standard deviations in Table 1.
Infrastructure Experiments are run using a Tesla V100-SXM2 GPU machine with 32GB RAM.
Code is written in PyTorch https://pytorch.org/ (Paszke et al., 2017).
C. A DS
Preprocessing Table 5 lists various statistics on the trafﬁc for the two ads datasets; we describe in this paragraph how datasets are processed.
ads-kaggle The Criteo Kaggle dataset is a reference dataset for CTR prediction and groups one week of log data.
The objective is to model the probability that a user will click on a given ad given his navigation context: p ( Click | Context ) .
Positives refer to displays which are clicked and negatives to displays not clicked.
For each datum, there are 13 continuous and 26 categorical features.
We divide the trafﬁc into two domains using feature number 30; for a given value for this categorical feature, all instances have a single missing numeric feature (feature number 5).
We then construct an artiﬁcial dataset simulating transfer between known and new users.
We process the original Criteo Kaggle dataset to have an equal number of source and target data.
We then perform train / test split on this dataset keeping 20 % of data for testing.
We used in our experiments the continuous features; to show the beneﬁt of modelling additional missing features, we extend the missing features list to features 1, 5, 6, 7, 11 and 12 by setting them to zero on the target domain.
After these operations, 6 features are missing and 7 are non-missing.
Preprocessing consists in normalizing continuous features using a log transform.
ads-real
The data collected in this dataset is similar to ads-kaggle .
We ﬁlter out non-clicks and the ﬁnal task is to model the sale probability for a clicked ad: p ( Sale | Click = 1 , Context ) .
Positives refer to clicked ads which lead to a sale; negatives to clicked ads which did not lead to a sale.
We use one week of sampled logs as a training set, and use the following day data as the test set.
This train / test deﬁnition is used so as to better correlate with the performance of a model used online.
We use typical features in production models on this dataset and focus on a sample of continuous features aggregated across user timelines describing the clicking and purchase behavior of a user.
In comparison to ads-kaggle more continuous features are used.
The count features can be User-centric i.e. describe the global activity of the user (number of clicks, displays, sales done globally across partners) or User-partner features i.e. describing the history of the user on the given partner (number of clicks, sales... on the partner).
The latter are missing for new users.
Counts are aggregated across varying windows of time and categories of partner catalog.
We bucketize these count features using log transforms and consider these as categorical features by one-hot encoding the different buckets based on the vocabulary ﬁle.
After processing the ﬁnal input size is 596 with 29 features used.
12 features are missing and 17 are non-missing.
[Table]
Evaluation On both datasets the train and test sets are ﬁxed.
We run each model ﬁve times with a different seed for initialization (using Xavier Uniform initialization) and report mean and standard deviations in Table 2.
Infrastructure For ads-real , models are trained using internal large-scale distributed infrastructure.
For ads-kaggle , we keep the same setting and code as in digits .
D I MPLEMENTATION DETAILS D. N EURAL N ET ARCHITECTURE D.1. DIGITS
We experiment with the ADV and OT versions of our imputation model.
For ADV models, we use the DANN model description in Ganin & Lempitsky (2015); for OT we use the DeepJDOT model description in Damodaran & Kellenberger (2018).
Both models can be considered as simpliﬁed instances of our corresponding ADV and OT imputation models when no imputation is performed.
Performance of the adaptation models is highly dependent on the NN architectures used for adaptation and classiﬁcation.
In order to perform fair comparisons and since our goal is to evaluate the potential of joint adaptation-imputation-classiﬁcation, we selected these architectures through pre- liminary tests and use them for both the ADV and OT models.
The two models are described below and illustrated in Figure 5.
• Feature extractors g 1 and g 2 consists of three convolutional layers with 5 × 5 kernel and 64 ﬁlters interleaved with max pooling layers with a stride of 2 and 2 × 2 kernel.
The ﬁnal layer hasﬁlters.
We use batch norm on convolutional layers and ReLU as an activation after max pooling.
As in Damodaran & Kellenberger (2018)
we ﬁnd that adding a sigmoid activation layer as ﬁnal activation is helpful.
• Classiﬁer f consists of two fully connected layers with 100 neurons with batch norm and ReLU activation followed by the ﬁnal softmax layer.
We add Dropout as an activation for the ﬁrst layer of the classiﬁer.
• Discriminator D 1 and D 2 is a single layer NN with 100 neurons, batch norm and ReLU followed by the ﬁnal softmax layer.
On USPS → MNIST and MNIST → USPS dataset we use a stronger discriminator network which consists of two fully connected layers with 512 neurons.
• Generator h consists of two fully connected layers with 512 neurons, batch norm and ReLU activation.
This architecture is used for ADV and OT imputation models.
In practice using wider and deeper networks increases classiﬁcation performance with the more complicated classiﬁcation tasks (SVHN → MNIST, MNIST → MNIST-M); in these cases we add an additional fully connected network with 512 neurons.
The ﬁnal activation function is a sigmoid.
We use the same architecture described above for full, missing and partial models to guarantee fair comparison.
As a side note, the input to the imputation model’s classiﬁer is twice bigger as in the standard adaptation models.
[Figure]
We experiment with ADV models only.
As input data is numeric and low dimensional, architectures are simpler than in digits .
ads-kaggle Our feature extractor is a three layered NN with 128 neurons and with a ﬁnal sigmoid activation.
The classiﬁer is taken to be a single layered NN with 128 neurons and a ﬁnal softmax layer.
Activations are taken to be ReLUs.
The domain discriminator is taken to be a two layered NN with 128 neurons and a ﬁnal softmax layer.
Finally the reconstructor is taken to be a two-layered NN with 256 neurons and ﬁnal sigmoid activation.
ads-real Input features after processing are fed directly into the feature extractors g 1 , g 2 consisting of two fully connected layers with 128 neurons.
The classiﬁer and discriminator is taken to be single-layered NN with 25 neurons.
The reconstructor is taken to be a two-layered NN withneurons.
Inner activations are taken to be ReLUs and the ﬁnal activation of the feature extractor is taken to be a sigmoid.
D. N ETWORK PARAMETERS D.2. H YPERPARAMETER TUNING
Tuning hyperparameters for UDA is tricky as we do not have access to the label in the target distribution and thus cannot choose parameters minimizing the target risk on a validation set.
Several papers set hyperparameters through reverse cross-validation (Ganin & Lempitsky, 2015).
Other approaches developed for model selection are based on risk surrogates obtained by estimating an approximation of the risk value on the source based on the similarity of source and target distributions (without the labels).
In the experiments, we used a recent estimator, Deep Embedded Validation (DEV)
(You et al., 2019) for tuning the initial learning rate; λ 1 and λ OT in OT .
For other parameters, we used heuristics and typical hyperparameter values from UDA papers (such as batch size) without further tuning.
We use a cross entropy link function on the source validation set; this value provides a proxy for the target test risk.
Using parameters from the original paper, this estimator helps select parameter ranges which perform reasonably well.
We keep the estimator unchanged for the full and missing adaptation models.
In the imputation case the discriminator used for computing importance sampling weights discriminates between [ (cid:101) x S 1 , (cid:101) x S 2 ] and [ (cid:101)
x T 1 , (cid:101) x T 2 ] (cf.
D 1 in Figure 1).
D.2. D IGITS
We ﬁnd that the results are highly dependent on the NN architecture and the training parameter setting.
In order to evaluate the gain obtained with Adaptation-Imputation , we use the same
NN architecture for all models ( ADV and OT ) but ﬁne tune the learning rates for each model using the
DEV estimator (other parameters do not have a signiﬁcant impact on the classiﬁcation performance).
ADV We use an adaptive approach as in Ganin & Lempitsky (2015) for decaying the learning rate lr and updating the gradient’s scale s between 0 and 1 for the domain discriminators.
We choose the decay values used in Ganin & Lempitsky (2015) ie.
s = 1 + exp( − 10 × p ) − 1 and lr =
lr i (1 + 10 × p ) 0 , 75 where p is ratio of current batches processed over the total number of batches to be processed without further tuning.
We tune the initial learning rate lr i , chosen in the range { 10 − 2 , 10 − 2 .
5 , 10 − 3 , 10 − 3 .
5 , 10 − 4 } following Section D.2.1.
In practise we take lr i = 10 − 2 for
ADV Adaptation-Imputation , Adaptation-Full , Adaptation-Partial and lr i = 10 − 2 .
5 for ADV Adaptation-Missing .
We use Adam (Kingma & Ba, 2014) as the optimizer with momentum parameters β 1 = 0 .
8 and β 2 = 0 .
and use the same decay strategy and initial learning rate for all components (feature extractor, classiﬁer, reconstructor).
Batch size is chosen to be 128; we see in practise that initializing the adaptation models with a source model with smaller batch size (such as 32) can be beneﬁcial.
OT We choose parameter λ 1 = λ OT = 0 .
1 in Equations 4 and 8 after tuning in the range { 10 − 1 , 10 − 2 , 10 − 3 } using DEV; the goal is indeed to down weight the contribution of alignment through Wasserstein distance which should be scaled compared to the classiﬁcation term.
Following Damodaran & Kellenberger (2018), batch size is taken to be 500 and we use EMD as Optimal Transport metric.
We initialize adaptation models with a source model in the ﬁrst 10 epochs and divide the initial learning rate by two as adaptation starts for non-imputation models.
For Adaptation-Imputation we follow a decaying strategy on the learning rate and on the adaptation weight as explained in the next item.
We choose lr i in the range { 10 − 2 , 10 − 2 .
5 , 10 − 3 , 10 − 3 .
5 , 10 − 4 } .
In practise we ﬁx lr i = 10 − 2 for all models.
Imputation parameters In practice we can deﬁne a weight for each term in Equation 2 and 4.
Further studies are conducted in Section 5.4 on weightings in Equation 2: in digits experiments we choose L 2 =
L MSE + L ADV for ADV and OT to reduce the burden of additional feature tuning.
For ADV model, we ﬁx parameters in Equation 4 to 1.
In the OT model, we vary λ 1 betweenand 0.1 and λ 2 between 0 and 1 following the same schedule as the gradient scale update for ADV models.
Although not necessary to achieve good mean values, this proves to reduce signiﬁcantly the variance.
D.2. A DS
We use an adaptive strategy for updating the gradient scale and the learning rate with the same parameters as in the digits dataset.
Optimizer is taken to be Adam.
Batch size is taken to be big so that target batches include sufﬁcient positive instances.
ads-kaggle The initial learning rate is chosen in the range { 10 − 4 , 10 − 5 , 10 − 6 , 10 − 7 } using
DEV and ﬁxed to be 10 − 6 for all models.
Batch size is taken to be 500 and we initialize models with a simple classiﬁcation loss for ﬁve epochs.
We run models for 50 epochs after which we notice that models reach a plateau.
We ﬁnd that adding a weighted MSE term allows to achieve higher stability (as measured by variance) as further studied in Section 5.4.
In a similar fashion to Pathak et al.
, we tune this weight in the range { 1 , 10 − 1 , 10 − 2 , 7 . 5 × 10 − 3 , 5 × 10 − 3 , 10 − 3 } .
We ﬁnd that 0 .
005 offers the best compromise between mean loss and variance.
Moreover on this dataset we use a faster decaying strategy for the discriminator’s D 2 and the reconstructor’s h learning rate, lr
= lr i (1 + 30 × p ) 0 , 75 to achieve higher stability in the training curves while the feature extractor g 1 , g 2 and D 1 ’s learning rate are unchanged.
ads-real
The initial learning rate is chosen in the range { 10 − 4 , 10 − 5 , 10 − 6 } and ﬁxed to be 10 − 6 for all models.
The learning rate is decayed with the same parameters as digits for all models.
We run models for ten epochs which provides a good trade-off between learning time and classiﬁcation performance.
Batch size is taken to be 500.
We choose L 2 =
L MSE + L ADV without further tuning; this simple weight achieves already good results.
E V ISUALIZATION OF FEATURES ’ DISTRIBUTION ON ADS - KAGGLE
The goal of this section is to visualize the domain shift existing in ads-kaggle pre-adaptation to show that the problem comprises both a domain shift and an imputation problem.
Table 6 reports mean and standard deviation on each feature’s value over the domain and Figure 6 plots the histogram of the normalized distributions for better visualization.
The y -axis is not normalized and corresponds to real counts.
We see that feature 5 is missing on the target domain and that distributions are different in shape, mean and variance on each domain
[Table] [Table] F E MBEDDING VISUALIZATION ON DIGITS
In this section we visualize the embeddings learned by the various models on digits by projecting the embeddings in a 2D space using t-SNE (the original embedding size being 2048).
Figure 7 represents the embeddings learned for ADV models on MNIST → MNIST-M. Figures 8 and 9 represent these embeddings for OT models respectively on MNIST → MNIST-M and MNIST → USPS.
On these ﬁgures, we see that imputation models generate feature representations that overlap better between source and target examples per class than the adaptation counterparts (although
Adaptation-Partial does a good job at overlapping feature representations).
This correlates with the accuracy performance on the test set.
Moreover we notice, as expected, that missing and partial adaptation networks perform badly compared to the full adaptation counterparts which justiﬁes the use of imputation model when confronted to missing data.
[Figure] [Figure]
[Figure] [Figure] [Figure]