W HY BE ADVERSARIAL ? L ET ’ S COOPERATE !: C OOPERATIVE D ATASET A LIGNMENT VIA JSD U PPER B OUND
Anonymous authors
Paper under double-blind review
A BSTRACT
Unsupervised dataset alignment estimates a transformation that maps two or more source domains to a shared aligned domain given only the domain datasets.
This task has many applications including generative modeling, unsupervised domain adaptation, and socially aware learning.
Most prior works use adversarial learning (i.e., min-max optimization), which can be challenging to optimize and evaluate.
A few recent works explore non-adversarial flow-based (i.e., invertible) approaches, but they lack a unified perspective.
Therefore, we propose to unify and generalize previous flow-based approaches under a single non-adversarial framework, which we prove is equivalent to minimizing an upper bound on the JensenShannon Divergence (JSD).
Importantly, our problem reduces to a min-min, i.e., cooperative, problem and can provide a natural evaluation metric for unsupervised dataset alignment.
We present empirical results of our framework on both simulated and real-world datasets to demonstrate the benefits of our approach.
I NTRODUCTION
In many cases, a practitioner has access to multiple related but distinct datasets such as agricultural measurements from two farms, experimental data collected in different months, or sales data before and after a major event.
Unsupervised dataset alignment (UDA) is the ML task aimed at aligning these related but distinct datasets in a shared space, which may be a latent space, without any pairing information between the two domains (i.e., unsupervised).
This task has many applications such as generative modeling (e.g., (Zhu et al., 2017)), unsupervised domain adaptation (e.g., (Grover et al., 2020; Hu et al., 2018)), batch effect mitigation in biology (e.g., (Haghverdi et al., 2018)), and fairness-aware learning (e.g., (Zemel et al., 2013)).
The most common approach for obtaining such alignment transformations stems from Generative
Adversarial Networks (GAN)(Goodfellow et al., 2014), which can be viewed as minimizing a lower bound on the Jensen-Shannon Divergence (JSD) between real and generated distributions.
The lower bound is tight if and only if the inner maximization is solved perfectly.
CycleGAN (Zhu et al., 2017) maps between two datasets via two GAN objectives between the two domains and a cycle consistency loss, which encourages approximate invertibility of the transformations.
However, adversarial learning can be challenging to optimize in practice (see e.g. (Lucic et al., 2018;
Kurach et al., 2019)) in part because of the competitive nature of the min-max optimization problem.
Perhaps more importantly, the research community only has reasonable GAN evaluation metrics for certain data types.
Specifically, the commonly accepted Frechet Inception Distance (FID) (Heusel et al., 2017) is only applicable to image or auditory data, which have standard powerful pretrained classifiers, and even the implementation of FID can have evaluation issues (Parmar et al., 2021).
No clear metrics exist for tabular data or non-perceptual data.
Recently, flow-based methods that leverage invertible models have been proposed for the UDA task (Grover et al., 2020; Usman et al., 2020).
AlignFlow (Grover et al., 2020) leverages invertible models to make the model cycle-consistent (i.e., invertible) by construction and introduce exact log-likelihood loss terms derived from standard flow-based generative models as a complementary loss terms to the adversarial loss terms.
Yet, AlignFlow still leverages adversarial learning.
Loglikelihood ratio minimizing flows (LRMF) (Usman et al., 2020) use invertible flow models and
[Figure]
density estimation for distribution alignment without adversarial learning and define a new metric based on the log-likelihood ratio.
However, LRMF depends heavily on the density model class and can only partially align datasets if the target distribution is not in the chosen density model class.
Additionally, the LRMF metric depends on this density model class and is only defined for two datasets.
Therefore, to provide an alternative to adversarial learning and generalize previous flow-based approaches we propose a unified non-adversarial UDA framework, which we prove is equivalent to minimizing an upper bound on the JSD.
Importantly, our problem reduces to a min-min, i.e., cooperative , problem and the JSD upper bound can provide a natural evaluation metric for UDA that can be applied in any domain.
Our framework requires two parts, the outer minimization requires an invertible model and the inner minimization requires a density model (e.g., Gaussian mixture models or normalizing flows (Dinh et al., 2017)).
We summarize our contributions as follows:
• We prove that a minimization problem over density models is an upper bound on a generalized version of JSD that allows for more than two distributions.
Importantly, we also theoretically quantify the bound gap and show that it can be made tight if the density model class is flexible enough.
• We use this JSD upper bound to derive a novel regularized loss function for UDA and explain its relationship to prior methods.
• We empirically demonstrate the benefits of our method compared to prior flow-based models on both simulated and real-world datasets.
Notation We will denote distributions as P X ( x ) where X is the corresponding random variable.
Invertible functions will be denoted by T ( · ) .
We will use X j ∼ P X j to denote the observed random variable from the j -th distribution.
We will use Z j ≜ T j ( X j ) ∼ P
Z j ≡ P T j ( X j ) to denote the latent random variable of the j -th distribution after applying T j to X j (and note that X j = T − 1 j ( Z j ) ).
We will denote the mixtures of these observed or latent distributions as P X mix ≜ (cid:80) j w j P X j and
P Z mix ≜ (cid:80) j w j P Z j , where w is a probability vector.
We denote KL divergence, entropy, and cross entropy as KL( · , · ) , H( · ) , and H c ( · , · ) , respectively, where KL( P, Q ) =
H c ( P, Q ) − H
( P ) .
R EGULARIZED A LIGNMENT U PPER B OUND L OSS
In this section, we will introduce our main theoretical result proving an upper bound on the generalized JSD divergence, deriving our loss function based on this, and then showing that minimizing this upper bound results in aligned distributions assuming large enough capacity of the model components.
Background: Normalizing Flows and Invertible Models Normalizing flows are generative models that have tractable distributions where exact density evaluations and efficient samplings are ensured (Kobyzev et al., 2021).
Such models leverage the change of variables formula to create an invertible mapping T such that P X ( x ) =
P Z ( T ( x ))
| J T ( x ) | where P Z is a known latent distribution and | J T ( x ) | is the absolute value of determinant of the Jacobian of the invertible map
T .
For sampling in distribution P X , one need to first sample from the latent distribution P Z
and then apply the inverse transform T − 1 .
Therefore the key challenge in designing invertible models is to have computationally efficient inverse evaluation and Jacobian determinant calculation.
Many approaches have been proposed by parameterizing mapping function T as deep neural networks including autoregressive structures (Kingma et al., 2016; Papamakarios et al., 2021), coupling layers (Dinh et al., 2017)(Kingma and Dhariwal, 2018), oridinary differential equations(Grathwohl et al., 2018), and invertible residual networks(Chen et al., 2019; Behrmann et al., 2019).
Flow models can be then learned efficiently by maximizing the likelihood for the given data.
Background: Generalized JSD
We remind the reader of the generalized Jensen-Shannon divergence for more than two distributions, where the standard JSD is recovered if w 1 = w 2 = 0 .
Definition 1 (Generalized Jensen-Shannon Divergence (GJSD)
(Lin, 1991)) .
Given k distributions { P X j } kj =1 and a corresponding probability weight vector w , the generalized Jensen-Shannon divergence is defined as (proof of equivalence in appendix):
[Equation]
2. GJSD U PPER B OUND
The goal of distribution alignment is to find a set of transformations { T j ( · ) } kj =1 (which will be invertible in our case) such that the latent distributions align, i.e., P T j ( X j ) =
P T j ′
X j ′ ) or equivalently
P Z j = P
Z j ′ for all j ̸ = j ′ .
Given the properties of divergences, this alignment will happen if and only if GJSD( P Z 1 , · · · , P Z k ) = 0 .
Thus, ideally, we would minimize GJSD directly with respect to T j , i.e.,
[Equation]
where T is a class of invertible functions.
However, we cannot evaluate the entropy terms in Eqn.
2 because we do not know the density of P X j ; we only have samples from P X j .
Therefore, we will upper bound the first entropy term in Eqn.
2 ( H (cid:0)(cid:80)
j w j P X j (cid:1) ) using an auxiliary density model and decompose the other entropy terms by leveraging the change of variables formula for invertible functions.
Theorem 1 (GJSD Upper Bound) .
Given an auxiliary density model class Q , we form a GJSD upper bound:
[Equation]
where the bound gap is exactly min Q ∈Q KL( P Z mix , Q ) .
Proof of Theorem 1.
For any Q ∈ Q , we have the following upper bound:
[Equation]
where the inequality is by the fact that KL divergence is non-negative and the bound gap is equal to KL( P Z mix , Q ) .
The Q that achieves the minimum in the upper bound is equivalent to the Q that minimizes the bound gap, i.e.,
[Equation]
The tightness of the bound depends on how well the class of density models Q (e.g., mixture models, normalizing flows, or autoregressive densities) can approximate P Z mix ; notably, the bound can be made tight if P Z mix ∈ Q .
Also, one key feature of this upper bound is that the cross entropy term can be evaluated using only samples from P X j and the transformations T j ,
i.e., H c ( P Z mix , Q ) =
(cid:80) j w j E P Xj [ − log Q
( T j ( x j ))] .
However, we still cannot evaluate the other entropy terms H( P Z j ) since we do not know the densities of P Z j (or P X j ).
Thus, we leverage the fact that the T j functions are invertible to define an entropy change of variables.
Lemma 2 (Entropy Change of Variables) .
Let X ∼ P X and Z ≜ T ( X ) ∼ P Z , where T is an invertible transformation.
The entropy of Z can be decomposed as follows:
[Equation]
where | J T ( x ) | is the determinant of the Jacobian of T .
The key insight from this lemma is that H( P X ) is a constant with respect to T and can thus be ignored when optimizing T , while E P X
[log | J T ( x ) | ] can be approximated using only samples from
P X .
2. A LIGNMENT U PPER B OUND (AUB)
Combining Theorem 1 and Lemma 2, we can arrive at our final objective function which is equivalent to minimizing an upper bound on the GJSD:
[Equation]
where the last term − (cid:80) j w j H( P X j ) is constant with respect to T j functions so they can be ignored.
We formally define this loss function as follows.
Definition 2 (Alignment Upper Bound Loss) .
Given k continuous distributions { P X j } kj =1 , a class of continuous distributions Q , and a probability weight vector w , the alignment upper bound loss is defined as follows:
[Equation]
where T j are invertible and | J T j ( x ) | is the absolute value of the Jacobian determinant.
Notice that this alignment loss can be seen as learning the best base distribution given fixed flow models T j .
We now consider the theoretical optimum if we optimize over all invertible functions.
Theorem 3 (Alignment at Global Minimum of L AUB ) .
If L AUB is minimized over the class of all invertible functions, a global minimum of L AUB implies that the latent distributions are aligned, i.e.,
P T j ( X j ) =
P T j ′
X j ′ ) for all j ̸ = j ′ .
Notably, this result holds regardless of Q .
Informally, this can be proved by showing that the problem decouples into separate normalizing flow losses where Q is the base distribution and the optimum is achieved only if P T j ( X j ) =
Q for all T j (formal proof in the appendix).
This alignment of the latent distributions also implies the translation between any of the observed component distributions.
The proof follows directly from Theoremand the change of variables formula.
Corollary 4 (Translation at Global Minimum of L AUB ) .
Similar to Theorem 3, a global minimum of L AUB implies translation between any component distributions using the inverses of T j , i.e.,
P T − 1 j ′ ( T j ( X j ))
=
P X j ′ for all j ̸ = j ′ .
Regularization via Transportation Cost While the alignment objective is the most challenging part of UDA, we suggest that regularization may also be useful for practical and stable alignment (or translation) between datasets because there are many optimal alignment solutions—even infinitely many in most cases (see appendix for two examples).
We alleviate this issue by adding expected transportation cost (usually squared Euclidean distance) as a regularization to our objective inspired by optimal transport (OT) concepts.
Definition 3 (Regularized Alignment Upper Bound Loss, RAUB) .
Given similar setup as in Def.and a transportation cost function c ( a, b ) ≥ 0 for transporting a point from a to b , the regularized alignment upper bound loss is defined as:
[Equation]
Pseudo-code for training our model with RAUB objective can be found in the appendix Section C.
R ELATIONSHIP TO P RIOR W ORKS
AlignFlow without adversarial terms is a special case AlignFlow (Grover et al., 2020) without adversarial loss terms is a special case of our method for two distributions where the density model class Q only contains the standard normal distribution (i.e., a singleton class) and no regularization is used (i.e., λ = 0 ).
Thus, AlignFlow can be viewed as initially optimizing a poor upper bound on
JSD; however, the JSD bound becomes tighter as training progresses because the latent distributions independently move towards the same normal distribution.
LRMF is special case with only one transformation Log-likelihood ratio minimizing flows (LRMF) (Usman et al., 2020) is also a special case of our method for only two distributions, where one transformation is fixed at the identity (i.e., T 2 = Id) and no regularization is applied (i.e., λ = 0 ).
While the final practical LRMF objective is a special case of ours, the theory is developed from a different but complementary perspective.
The LRMF metric developed requires an assumption about a given density model class, which enables a zero point (or absolute value) of the metric to be estimated but requires fitting extra domain density models.
Usman et al.
(2020) also do not uncover the connection of the objective as an upper bound on JSD regardless of the density model class.
Additionally, to ensure alignment, LRMF requires that the density model class includes the true target distribution because only one invertible transform is used, while our approach can theoretically align even if the shared density model class is weak (see Theorem 3 and our simulated experiments).
See Fig.
2 for a comparison of our approach to prior works.
[Figure]
Cooperative networks versus adversarial networks Analogous to the generator G and the discriminator D in adversarial learning, our framework has two main networks, T j and Q z .
We can use any invertible function for T j (e.g., coupling-based flows (Dinh et al., 2017), neural ODE flows (Grathwohl et al., 2018), or residual flows (Chen et al., 2019)) and any (approximate) density models for Q z (e.g., kernel densities (in low dimensions), mixture models, autoregressive densities (Salimans et al., 2017), normalizing flows (Kingma and Dhariwal, 2018), or even VAEs (Kingma and
Welling, 2019)).
Thus, our framework has similar modularity compared to adversarial approaches.
In contrast, we have a min-min, i.e., cooperative, optimization problem, but our transformations must be invertible.
As another difference, the inner optimization problem may be more challenging (i.e., fitting a density model Q z may be more difficult than fitting an auxiliary discriminator D ), but the overall min-min is likely to be more stable than the min-max problem—specifically, note that our problem can align even if the density model Q z is not optimal (see Theorem 3).
We expect some of these limitations to be alleviated as new invertible models and density models are continually being developed.
Therefore, the proposed approach provides a fundamental alternative to adversarial learning with different strengths and weaknesses.
E XPERIMENT 4. 2D DATASET COMPARISON WITH RELATED WORKS
We compare our method with related works in order to illustrate the limitations of prior methods and how our method overcomes these limitations.
All implementation details for the 2D dataset are available in the appendix.
Single T vs. Double T (LRMF vs. Ours)
We first compare our method with LRMF (Usman et al., 2020) method.
We construct the experiment to have the task:
Translation between the two half-circled distributions: X 1 and X 2 in the moons dataset.
In this example, we made two models, one with LRMF setup and one with our AUB setup.
As illustrated in Fig.
3, the LRMF method fails to transform between X 1 and X 2 .
Even though Q can model well enough for T 1 ( X 1 ) , Q can only model the mean and variance of X 2 which is obviously not informative enough.
Therefore, the
LRMF fails to transform between two datasets.
While in the AUB setup, both T 1 ( X 1 ) and T 2 ( X 2 ) are modeled to the same distribution which Q can be learned to fit with high likelihood which leads to better translation results.
In conclusion, the performance of the LRMF model is limited by the power of the density model Q which means if Q fails to model one of the domain distribution in high likelihood, data alignment cannot be achieved with good performances.
Simple Fixed Q vs. Learnable Q (AlignFlow vs. Ours)
Next we compare our method with
AlignFlow(Grover et al., 2020;
Hu et al., 2018) setup.
We construct the experiment to have the task:
Transform between the two random patterns X 1 and X 2 from the randomly generated datasets.
Again, we made two models with AlignFlow and our AUB setups respectively.
As illustrated in
Fig.
3, the AlignFlow method fails to transform between X 1 and X 2 , because the transformed dataset
T 1 ( X 1 ) and T 2 ( X 2 ) failed to reach the normal distribution Q .
While in the AUB setup, the density model Q is learned to help fit the transformed distributions T 1 ( X 1 ) and T 2 ( X 2 ) , which allows them to be aligned with each other easier.
In conclusion, the performance of the AlignFlow model is limited by the performances of the invertible functions.
Regularized vs. Un-regularized (Some prior works vs. Ours)
We finally show the importance of the regularization term.
We construct the experiment to have the task: Transform between two concentric circles with the same mean but slighted different radius.
In this example, we make two models with our AUB approach, but one with regularization and one without.
As illustrated in the
Figure 4, both models are able to transform between two distributions.
However, the transformation pattern is not natural in terms of the movement of each point.
Each pair created by the unregularized model has bigger transportation cost compared to the pairs created by the regularized model.
Therefore, we suggest that by adding transportation cost, the resulting transformation between samples will be closer to the identity function and therefore more stable.
4. R EAL - WORLD DATASETS
Metrics In this subsection, we use two metrics to measure the overall model performances:
Frechet Inception Distance (FID) (Heusel et al., 2017) implemented by (Seitzer, 2020) and our metric Alignment Upper Bound (AUB).
[Figure]
[Figure]
Unsupervised domain translation We first perform three image translation tasks on MNIST dataset; more specifically, we are flipping digit images in ”0/1”, ”0/2” , and ”1/2” pairs shown in Fig.
We use RealNVP invertible models for all translation maps T j , as well as the density model Q .
We evaluate our model performances in Table 1 along with baseline models: LRMF and
MLE version of AlignFlow.
Note that all methods are flow-based models and thus images translated back to original domain are exactly the same, which implies exact cycle consistency.
As represented in Table.
both of our approaches outperform the baseline models in terms of FID and AUB.
The lower score in AUB indicates our model has the tighter bound than baseline model.
This implies the statistical distance between the shared distribution { Z j } kj =1 and our density model
Q is smaller than baselines, meaning our model has the better dataset alignment performance.
We believe this result comes from our model setup, i.e., a learnable density model and domain specific invertible transformation functions with shared space.
Specifically, AlignFlow with a fixed standard normal distribution as their Q obtains worse AUB because the Q is not powerful enough to model the complex shared space trained from the real world dataset.
On the other hand, LRMF shows the lack of stability when trained with relatively simple models that we are using, i.e., RealNVP T and
Real NVP Q .
This is because the transformation function and the density model should be able to directly model the complex real distribution in their setup.
By comparing our approach with and without transportation cost, we observe introducing transportation cost yields insignificant decrease in AUB and an increase in FID.
This indicates the transportation cost regularizes our method well without degeneration, so that our method can find better solution (which is theoretically unique).
The impressive performance of our method in AUB shows a consistent pattern in FID score as well.
Across entire translation cases, our method shows the better performance in terms of FID.
This result can be intuitively understood by Fig.
AlignFlow shows less stable translation results than our method especially for translating to digit ‘2’ from digits ‘0’ and ‘1’.
We believe this phenomenon comes from the lack of expressivity of their Q model.
On the other hand, our model shows stable results across all translation cases, which are also quantitatively verified via the lower FID score.
Also, from the comparison on the latent representations (second column of each macro column) of ours with and without transportation cost, we can observe that the latent representation of the λ =setup contains numerous excessive values.
This implies adding transportation cost ( lambda = 1 ) prevents the excessive movement in the latent space, thus, our model can find a simpler and smoother solution during the training.
[Figure]
[Table] [Table]
Multi-domain translation To illustrate that our method can be easily scaled to more domain distributions, we present qualitative examples of translating between every digit and every other digit for MNIST in Fig.
6 with performances in Table 2.
To show the effect of transportation cost proposed in our paper, we also compare our model with and without transportation cost.
Note that we omit LRMF in this experiment because the multi-domain situation is hard to deal with for LRMF setup due to LRMF’s two distribution setup and assymetric model structure.
As shown in Table 2, our approach with and without transportation cost shows better performance in terms of FID and AUB than AlignFlow since our learnable density estimator can model more complex distribution than fixed simple density model in AlignFlow.
In other words, the shared space of AlignFlow is limited because of the fixed simple density model.
Note that the gap in the
AUB score between AlignFow and our model is larger in this multi-domain setting (Table 2) than in the two domain case (Table 1).
We hypothesize that this is because the shared density model across many domains needs to be more complex than for only two domains and thus, our learnable
Q can create a better upper bound to optimize than AlignFlow.
In this experiment, while ours with transportation cost performs slightly worse, the results are comparable and it does provide a natural regularization that ensures a theoretically identifiable solution.
The superiority of our method compared to AlignFlow in multiple-domain translation can also be verifed through qualitative comparisons in Fig.
The leftmost column is an input image and the second and third macro columns are the results from ours and AlignFlow.
by forwarding a given k -th latent T k ( x k ) into 10 inverse transformation functions, respectively.
It is easy to observe that ours has more clear results in most cases than baseline.
Moreover, our model shows better performance in maintaining the original identity (e.g., width and type of a stroke) than the baseline, as seen in third, fourth and ninth rows.
This is because we jointly train our transformation functions with a learnable density model, while AlignFlow independently train their transformation functions.
This benefit of our approach may be crucial for other datasets such as human faces (Choi et al., 2018) where maintaining the original identity is important.
[Figure]
[Table]
D ISCUSSION AND C ONCLUSION
Transfer Learning Capability
The performance of our model can be improved by adapting pretrained density models.
Fig.
7(a) shows preliminary results.
Specifically, a MLE version of AlignFlow model may still struggle to translate between USPS and MNIST dataset after learning forepochs while ours can show qualitative results after only 1 epoch if we are using pretrained powerful density model for Q Salimans et al.
Generative Tasks
Our model is also capable of generating samples in each domain.
One needs to sample from the density model Q to have a latent image z first, and then forward to the inverse function T − 1 j to get the image sampled in j th domain.
Examples of generated images for each domain in MNIST data are shown in Fig. 7(b).
The quality of the generative result also reflects the tightness of the bound between the latent space and the latent density model as illustrated in Eqn. 4.
[Figure]
-domain translation tasks in subsection 4.2.
In this paper, we proposed a novel upper bound on the generalized JSD that leads to a theoretically grounded alignment loss function.
We then show that this framework unifies previous flow-based approaches to dataset alignment and demonstrate the benefits of our approach compared to prior flow-based methods.
Additionally, we show that our framework could enable the use of pre-trained density models which would enable a type of transfer learning for distribution alignment.
More broadly, we expect that our AUB metric can be useful as a domain-agnostic metric for comparing dataset alignment methods beyond images.
An alignment metric that is not tied to a particular pretrained model (as for FID) or to a particular data type will be critical for systematic progress in unsupervised dataset alignment.
We hope this paper provides one step in that direction.
A P ROOFS
Proof of Equivalence in Def. 1.
While the proof of the equivalence is well-known
[Equation]
we reproduce here for completeness.
As a reminder
[Equation]
the KL divergence is defined as:
[Equation]
where H c ( · , · ) denotes the cross entropy and H( · ) denotes entropy.
Given this
[Equation]
we can now easily derive the equivalence:
[Equation]
Proof of Lemma 2.
First
[Equation]
we note the following fact from the standard change of variables formula:
[Equation]
We can now derive our result using the change of variables for expectations (i
[Equation]
e., LOTUS) and the probability change of variables from above:
[Equation]
Proof of Theorem 3.
Given any fixed Q
[Equation]
minimizing L AUB decouples into minimizing separate normalizing flow losses where Q is the base distribution.
For each normalizing flow
[Equation]
there exists an invertible T j such that T j ( X j ) ∼ Q
[Equation]
and this achieves the minimum value of L AUB .
More formally,
[Equation]
Given that KL( P, Q ) ≥ 0 and equal to 0 if and only if P = Q , the global minimum is achieved only if P T j ( X j )
=
Q, ∀ j and there exist such invertible functions (e.g., the optimal Monge map between
P X j and Q for squared Euclidean transportation cost (Peyr´e and Cuturi, 2019)).
Additionally, the optimal value is (cid:80) j w j H( P X j ) , which is constant with respect to the T j transformations.
B E XAMPLES OF N ON -U NIQUE A LIGNMENT S OLUTIONS B. G AUSSIAN E XAMPLE
Suppose the component distributions are normal distributions, i.e., X 1 ∼ N ( µ 1 , I ) and X 2 ∼
N ( µ 2 , I ) , and for even greater simplicity, we assume T 2 is the identity, i.e., T 2 ( x ) = x .
Then, a global optimal solution could be T 1 ( x ) =
U ( x − µ 1 + µ 2 ) for any orthogonal matrix U , i.e., there are infinitely many invertible functions that align the distributions.
Note that this lack of unique solutions is not restricted to orthogonal rotations (see appendix for a more complex example).
B. C OMPLEX E XAMPLE
Consider the 1D case where Q only contains the uniform distribution.
Thus, T 1 and T 2 must map their distributions to the uniform distribution for alignment.
One solution would be that T 1 =
Fand T 2 = F 2 where F 1 and F 2 are the CDFs of P X 1 and P X 2 .
Yet, there are infinitely many other possible solutions.
Consider an invertible function that subdivides the unit interval into an arbitrarily large number of equal length intervals and then shuffles these intervals with a fixed arbitrary permutation.
More formally, we could define this as:
[Equation]
where π ( · ) is a permutation of the integers 1 to m .
Given this, then other optimal solutions could be
T 1 = S m,π ◦ F 1 and T 2 = F 2 for any m > 1 and any permutation π .
This idea could be generalized to higher dimensions as well by mapping to the multivariate uniform distribution and subdividing the unit hypercube similarly.
C A LGORITHM
We summarize our computation of regularized alignment upper bound in Alg. 1.
Output do do ϕ θ j j do do end
D D ETAILED P ARAMETERS U SED IN 2D D ATASET E XPERIMENT D. LRMF VS . O URS E XPERIMENT
• T for LRMF setup: T 1 : 8 channel-wise mask for Real-NVP model with s and t derived from 64 hidden channels of fully connected networks.
T 2 : Identity function.
• T for RAUB setup: T 1 and T 1 : 8 channel-wise mask for Real-NVP model with s and t derived from 64 hidden channels of fully connected networks.
Regularization coefficient λ =• Q for both: A single Gaussian distribution with trainable mean and trainable variances.
D. A LIGNFLOW VS . O URS E XPERIMENT
• T for both: 2 channel-wise mask for RealNVP model with s and t derived from 8 hidden channels of fully connected networks.
• Q for Alignflow setup: A single fixed normal distribution.
• Q for RAUB setup: A learnable mixture of Gaussian with 3 components.
Regularization coefficient λ =
D. R EGULARIZED VS . U NREGULARIZED E XPERIMENT
T for both: 8 channel-wise mask for RealNVP model with s and t derived from 64 hidden channels of fully connected networks.
Q for both: A learnable mixture of Gaussian with 2 components.
λ for unregularized Experiment : λ =4.
λ for regularized Experiment : λ =
E I NTERPOLATION OVER LATENT SPACE
We use the same model used in the ”Multi-domain translation” part in section 4.2 and perform interpolation in the latent space.
We first randomly select two distinct real images in one domain (in this case two 0s), and do a linear interpolation of the selected two images in the latent space.
Then we translate all the interpolated images (including the two selected images) to all of the domains to generate ”translated-interpolated” images—i.e., the corresponding interpolations in each of the domains.
As shown in Fig.
8, all ”translated-interpolated” results can preserve the trend of the stroke width of the digits from the original interpolated domain.
These results suggest that our approach aligns the domains so that some latent space directions have similar semantic meaning for all domains.
F T ABULAR / STRUCTURED DATA EXPERIMENT
We follow the same preprocessing for four UCI tabular datasets (Dua and Graff, 2017) from the MAF paper (Papamakarios et al., 2017), specifically MINIBOONE, GAS, HEPMASS, and POWER.
In order to obtain a binary label for creating domains, we choose the last input feature for each dataset and discretize it based on whether it is higher or lower than the median value, which ensures the datasets are of equal size.
We use the MLE version of AlignFlow as a baseline to compare with our model and use the AUB score as the metric for comparison.
As shown in the Table.
3, our method shows better performance with a large gap across all comparisons.
This provides strong evidence that our proposed method (trained with learnable Q ) better aligns the two domains than AlignFlow (trained with fixed Q ).
It is worth emphasizing that there is no natural metric for evaluating GAN (Goodfellow et al., 2014) in tabular dataset.
We believe this demonstrates one of the key benefits of our proposed method over
GAN-based alignment methods.
G M ULTI - TRANSLATION TASKS IN C ELEB A DATASET
In order to verify our proposed method performs well in a more complex setup than MNIST, we conducted experiments on the CelebA (Liu et al., 2015) dataset.
We first create three domains (Black Hair, Blond Hair and Brown Hair) by using the hair color attribute.
We further center crop
[Table]
the images to have the size 148 × 148 then resize them to 64 × 64.
For the baseline model, we again uses MLE version of AlignFlow.
We use both FID and AUB as a metric to compare the performances.
As shown in the Table 4, our proposed method demonstrates better performance in almost all experiments in terms of both FID and AUB.
This consistently validates that the flexibility of Q in our framework has clear advantages over the fixed Q in AlignFlow for the domain alignment task.
Intuitively, we think the tighter upper bound from our proposed method plays an important role in aligning distributions in the latent space.
We also provide qualitative results in Fig. 9.
Compared to the AlignFlow (MLE) baseline, our model performs significantly better in translating between selected attributes.
In particular, while our approach maintains a valid face structure during translation, AlignFlow is unable to maintain a valid face structure.
Note that absolute image quality is not the focus of our paper as we use relatively simple flow models, but rather the comparison to prior methods and to show the feasibility of our approach in higher dimensions.
We expect better qualitative results could be achieved using more advanced flow models such as GLOW, Residual Flows, Flow++, etc.
but given that the scope of our work which focuses on the theoretical and foundational approach, we leave this to future work.
[Table]
[Figure]
[Equation]
[Figure]