Original: Why be adversarial? Let’s cooperate!: Cooperative Dataset Alignment via JSD Upper Bound
Revised: Why be adversarial? Let’s cooperate!: Cooperative Dataset Alignment via JSD Upper Bound

Original: Abstract
Revised: Abstract

Original: Unsupervised dataset alignment estimates a transformation that maps two or more source domains to a shared aligned domain given only the domain datasets.
Revised: Unsupervised dataset alignment estimates a transformation that maps two or more source domains to a shared aligned domain given only the domain datasets.

Original: This task has many applications including generative modeling, unsupervised domain adaptation, and socially aware learning.
Revised: This task has many applications including generative modeling, unsupervised domain adaptation, and socially aware learning.

Original: Most prior works use adversarial learning (i.e., min-max optimization), which can be challenging to optimize and evaluate.
Revised: Most prior works use adversarial learning (i.e., min-max optimization), which can be challenging to optimize and evaluate.

Original: A few recent works explore non-adversarial ﬂow-based (i.e., invertible) approaches, but they lack a uniﬁed perspective.
Revised: A few recent works explore non-adversarial ﬂow-based (i.e., invertible) approaches, but they lack a uniﬁed perspective.

Original: Therefore, we propose to unify and generalize previous ﬂow-based approaches under a single non-adversarial framework, which we prove is equivalent to minimizing an upper bound on the Jensen-Shannon Divergence (JSD).
Revised: Therefore, we propose to unify and generalize previous ﬂow-based approaches under a single non-adversarial framework, which we prove is equivalent to minimizing an upper bound on the Jensen-Shannon Divergence (JSD).

Original: Importantly, our problem reduces to a min-min, i.e., cooperative, problem and can provide a natural evaluation metric for unsupervised dataset alignment.
Revised: Importantly, our problem reduces to a min-min, i.e., cooperative, problem and can provide a natural evaluation metric for unsupervised dataset alignment.

Original: We present preliminary results of ourproposed framework on simulated and real-world data.
Revised: We present preliminary results of our framework on simulated and real-world data.

Original: 1. Introduction
Revised: 1. Introduction

Original: In many cases, a practitioner has access to multiple related but distinct datasets such as agricultural measurements from two farms, experimental data collected in different months, or sales data before and after a major event.
Revised: In many cases, a practitioner has access to multiple related but distinct datasets such as agricultural measurements from two farms, experimental data collected in different months or sales data before and after a major event.

Original: Unsupervised dataset alignment (UDA) is the ML task aimed at aligning these related but distinct datasets in a shared space, which may be a latent space, without any pairing information between the two domains (i.e., unsupervised).
Revised: Unsupervised dataset alignment (UDA) is the ML task aimed at aligning these related but distinct datasets in a shared space, which may be a latent space, without any pairing information between the two domains (i.e., unsupervised).

Original: This task has many applications such as generative modeling (e.g., (Zhu et al., 2017)), unsupervised domain adaptation (e.g., (Grover et al., 2020; Hu et al., 2018)), batch effect mitigation in biology (e.g., (Haghverdi et al., 2018)), and fairness-aware learning (e.g., (Zemel et al., 2013)).
Revised: This task has many applications such as generative modeling (e.g., (Zhu et al., 2017)), unsupervised domain adaptation (e.g., (Grover et al., 2020; Hu et al., 2018)), batch effect mitigation in biology (e.g., (Haghverdi et al., 2018)), and fairness-aware learning (e.g., (Zemel et al., 2013)).

Original: The most common approach for obtaining such alignment transformations stems from Generative Adversarial
Revised: The most common approach for obtaining such alignment transformations stems from Generative Adversarial

Original: Networks (GAN) (Goodfellow et al., 2014), which can be viewed as minimizing a lower bound on the JensenShannon Divergence (JSD) between real and generated distributions.
Revised: Networks (GAN) (Goodfellow et al., 2014), which can be viewed as minimizing a lower bound on the JensenShannon Divergence (JSD) between real and generated distributions.

Original: The lower bound is tight if and only if the inner maximization is solved perfectly.
Revised: The lower bound is tight if and only if the inner maximization is solved perfectly.

Original: CycleGAN (Zhu et al., 2017) maps between two datasets via two GAN objectives between the two domains and a cycle consistency loss, which encourages approximate invertibility of the transformations.
Revised: CycleGAN (Zhu et al., 2017) maps between two datasets via two GAN objectives between the two domains and a cycle consistency loss, which encourages approximate invertibility of the transformations.

Original: However, adversarial learning can be quite challenging in practice (see e.g. (Lucic et al., 2018; Kurach et al., 2019))
Revised: However, adversarial learning can be quite challenging in practice (see e.g. (Lucic et al., 2018; Kurach et al., 2019))

Original: because of the competitive nature of the minmax optimization problem.
Revised: because of the competitive nature of the minmax optimization problem.

Original: Also, the research community only has reasonable model evaluation metrics for certain data types.
Revised: Also, the research community only has reasonable model evaluation metrics for certain data types.

Original: Speciﬁcally, the commonly accepted Frechet
Revised: Speciﬁcally, the commonly accepted Frechet

Original: Inception Distance (FID) (Heusel et al., 2017) is only applicable to image or auditory data, which have standard powerful pretrained classiﬁers, and even the implementation of
Revised: Inception Distance (FID) (Heusel et al., 2017) is only applicable to image or auditory data, which have standard powerful pretrained classiﬁers, and even the implementation of

Original: FID can  evaluation issues (Parmar et al., 2021).
Revised: FID can have evaluation issues (Parmar et al., 2021).

Original: No clear metrics exist for tabular data or non-perceptual data.
Revised: No clear metrics exist for tabular data or non-perceptual data.

Original: Recently, ﬂow-based methods that leverage invertible models have been proposed for the UDA task (Grover et al., 2020; Usman et al., 2020).
Revised: Recently, ﬂow-based methods that leverage invertible models have been proposed for the UDA task (Grover et al., 2020; Usman et al., 2020).

Original: AlignFlow (Grover et al., 2020) leverages invertible models to make the model cycleconsistent (i.e., invertible) by construction and introduce exact log-likelihood loss terms derived from standard ﬂowbased generative models as a complementary loss terms to the adversarial loss terms.
Revised: AlignFlow (Grover et al., 2020) leverages invertible models to make the model cycleconsistent (i.e., invertible) by construction and introduce exact log-likelihood loss terms derived from standard ﬂowbased generative models as a complementary loss terms to the adversarial loss terms.

Original: Yet, AlignFlow still leverages adversarial learning and does not provide a general evaluation metric.
Revised: Yet, AlignFlow still leverages adversarial learning and does not provide a general evaluation metric.

Original: Log-likelihood ratio minimizing ﬂows (LRMF) (Usman et al., 2020) use invertible ﬂow models and density estimation to avoid adversarial learning altogether and deﬁne a new metric based on the log-likelihood ratio.
Revised: Log-likelihood ratio minimizing ﬂows (LRMF) (Usman et al., 2020) use invertible ﬂow models and density estimation to avoid adversarial learning altogether and deﬁne a new metric based on the log-likelihood ratio.

Original: However, LRMF depends heavily on the density model class and can only partially align datasets if the target distribution is not in the chosen density model class.
Revised: However, LRMF depends heavily on the density model class and can only partially align datasets if the target distribution is not in the chosen density model class.

Original: Additionally, the LRMF metric depends on this density model class and is only deﬁned for two datasets.
Revised: Additionally, the LRMF metric depends on this density model class and is only deﬁned for two datasets.

Original: Therefore, to avoid challenging adversarial learning and generalize previous ﬂow-based approaches, we propose a uniﬁed non-adversarial UDA framework, which we prove is equivalent to minimizing an upper bound on the JSD.
Revised: Therefore, to avoid challenging adversarial learning and generalize previous ﬂow-based approaches, we propose a uniﬁed non-adversarial UDA framework, which we prove is equivalent to minimizing an upper bound on the JSD.

Original: Importantly, our problem reduces to a min-min, i.e., cooperative , problem, and the JSD upper bound can provide a natural evaluation metric for UDA that can be applied in any domain.
Revised: Importantly, our problem reduces to a min-min, i.e., cooperative , problem, and the JSD upper bound can provide a natural evaluation metric for UDA that can be applied in any domain.

Original: Our framework requires two parts, the outer min- imization requires an invertible model and the inner minimization requires a density model (e.g., Gaussian mixture models or normalizing ﬂows (Dinh et al., 2017)).
Revised: Our framework requires two parts, the outer minimization requires an invertible model and the inner minimization requires a density model (e.g., Gaussian mixture models or normalizing ﬂows (Dinh et al., 2017)).

Original: We summarize our contributions as follows:
Revised: We summarize our contributions as follows:

Original: • We prove that a minimization problem over density models is an upper bound on a generalized version of JSD that allows for more than two distributions.
Revised: • We prove that a minimization problem over density models is an upper bound on a generalized version of JSD that allows for more than two distributions.

Original: Importantly, we also theoretically quantify the bound gap and show that it can be made tight if the density model class is ﬂexible enough.
Revised: Importantly, we also theoretically quantify the bound gap and show that it can be made tight if the density model class is ﬂexible enough.

Original: • We use this JSD upper bound to derive a novel regularized loss function for UDA and explain its relationship to prior methods.
Revised: • We use this JSD upper bound to derive a novel regularized loss function for UDA and explain its relationship to prior methods.

Original: • We demonstrate the feasibility of our method on simulated and real-world data.
Revised: • We demonstrate the feasibility of our method on simulated and real-world data.

Original: Notation We will denote distributions as P X ( x ) where
Revised: Notation We will denote distributions as P X ( x ) where

Original: X is the corresponding random variable.
Revised: X is the corresponding random variable.

Original: Invertible functions will be denoted by T ( · ) .
Revised: Invertible functions will be denoted by T ( · ) .

Original: We will use X j ∼ P X j to denote the observed random variable from the j -th distribution.
Revised: We will use X j ∼ P X j to denote the observed random variable from the j -th distribution.

Original: We will use Z j (cid:44) T j ( X j )
Revised: We will use Z j (cid:44) T j ( X j )

Original: ∼ P
Revised: ∼ P

Original: Z
Revised: Z

Original: j ≡ P T j ( X j ) to denote the latent random variable of the j -th distribution after applying T j to X j (and note that X j = T − 1 j ( Z j ) ).
Revised: j ≡ P T j ( X j ) to denote the latent random variable of the j -th distribution after applying T j to X j (and note that X j = T − 1 j ( Z j ) ).

Original: We will denote the mixtures of these observed or latent distributions as P X mix (cid:44) (cid:80) j w j P X j and P Z mix (cid:44) (cid:80)
Revised: We will denote the mixtures of these observed or latent distributions as P X mix (cid:44) (cid:80) j w j P X j and P Z mix (cid:44) (cid:80)

Original: j [Equation] where w is a probability vector.
Revised: j w j P Z j , where w is a probability vector.

Original: We denote KL divergence, entropy, and cross entropy as KL( · , · ) , H( · ) , and H c ( · , · ) , respectively, where KL( P, Q ) =
Revised: We denote KL divergence, entropy, and cross entropy as KL( · , · ) , H( · ) , and H c ( · , · ) , respectively, where KL( P, Q ) =

Original: H c ( P, Q ) − H
Revised: H c ( P, Q ) − H

Original: ( P ) .
Revised: ( P ) .

Original: 2. Regularized Alignment Upper Bound Loss
Revised: 2. Regularized Alignment Upper Bound Loss

Original: We ﬁrst remind the reader of the generalized JensenShannon divergence for more than two distributions, where the standard JSD is recovered if w 1 = w 2 = 0 .
Revised: We ﬁrst remind the reader of the generalized JensenShannon divergence for more than two distributions, where the standard JSD is recovered if w 1 = w 2 = 0 .

Original: Deﬁnition 1 (Generalized Jensen-Shannon Divergence (GJSD)
Revised: Deﬁnition 1 (Generalized Jensen-Shannon Divergence (GJSD)

Original: (Lin, 1991)) .
Revised: (Lin, 1991)) .

Original: Given k distributions { P X j } kj =1 and a corresponding probability weight vector w , the generalized Jensen-Shannon divergence is deﬁned as (proof of equivalence in appendix):
Revised: Given k distributions { P X j } kj =1 and a corresponding probability weight vector w , the generalized Jensen-Shannon divergence is deﬁned as (proof of equivalence in appendix):

Original: [Equation]
Revised: [Equation]

Original: The goal of distribution alignment is to ﬁnd a set of transformations { T j ( · ) } kj =1 (which will be invertible in our case) such that the latent distributions align, i.e., P T j ( X j ) =
Revised: The goal of distribution alignment is to ﬁnd a set of transformations { T j ( · ) } kj =1 (which will be invertible in our case) such that the latent distributions align, i.e., P T j ( X j ) =

Original: P T j (cid:48)
Revised: P T j (cid:48)

Original: ( X j (cid:48) ) or equivalently P Z j = P Z j (cid:48) for all j (cid:54) =
Revised: ( X j (cid:48) ) or equivalently P Z j = P Z j (cid:48) for all j (cid:54) =

Original: j (cid:48) .
Revised: j (cid:48) .

Original: Given the properties of divergences, this alignment will happen if and only if GJSD( P Z 1 , · · · , P Z k ) = 0 .
Revised: Given the properties of divergences, this alignment will happen if and only if GJSD( P Z 1 , · · · , P Z k ) = 0 .

Original: Thus, ideally, we would minimize GJSD directly with respect to T j , i.e.,
Revised: Thus, ideally, we would minimize GJSD directly with respect to T j , i.e.,

Revised: [Equation]

Revised: , ··· ,T k

Original: [Equation]
Revised: [Equation]

Original: where T is a class of invertible functions.
Revised: where T is a class of invertible functions.

Original: 2.1. GJSD Upper Bound
Revised: 2.1. GJSD Upper Bound

Original: However, we cannot evaluate the entropy terms in Eqn.because we do not know the density of P X j ; we only have samples from P X j .
Revised: However, we cannot evaluate the entropy terms in Eqn.because we do not know the density of P X j ; we only have samples from P X j .

Original: Therefore, we will upper bound the ﬁrst entropy term in Eqn.
Revised: Therefore, we will upper bound the ﬁrst entropy term in Eqn.

Original: 3 ( H (cid:0)(cid:80)
Revised: 2 ( H (cid:0)(cid:80)

Original: j w j P X j (cid:1) ) using an auxiliary density model and decompose the other entropy terms by leveraging the change of variables formula for invertible functions.
Revised: j w j P X j (cid:1) ) using an auxiliary density model and decompose the other entropy terms by leveraging the change of variables formula for invertible functions.

Original: Theorem 1 (GJSD Upper Bound) .
Revised: Theorem 1 (GJSD Upper Bound) .

Original: Given an auxiliary density model class Q , we form a GJSD upper bound:
Revised: Given an auxiliary density model class Q , we form a GJSD upper bound:

Original: [Equation]
Revised: [Equation]

Original: where the bound gap is exactly min Q ∈Q KL( P Z mix , Q ) .
Revised: where the bound gap is exactly min Q ∈Q KL( P Z mix , Q ) .

Original: Proof of Theorem 1.
Revised: Proof of Theorem 1.

Original: For any Q ∈ Q , we have the following upper bound:
Revised: For any Q ∈ Q , we have the following upper bound:

Original: [Equation]
Revised: [Equation]

Original: where the inequality is by the fact that KL divergence is non-negative and the bound gap is equal to KL( P Z mix , Q ) .
Revised: where the inequality is by the fact that KL divergence is non-negative and the bound gap is equal to KL( P Z mix , Q ) .

Original: The Q that achieves the minimum in the upper bound is equivalent to the Q that minimizes the bound gap, i.e., 
Revised: The Q that achieves the minimum in the upper bound is equivalent to the Q that minimizes the bound gap, i.e., Q

Original: [Equation]
Revised: [Equation]

Original: The tightness of the bound depends on how well the class of density models Q (e.g., mixture models, normalizing ﬂows, or autoregressive densities) can approximate P Z mix ; notably, the bound can be made tight if P Z mix ∈ Q .
Revised: The tightness of the bound depends on how well the class of density models Q (e [Equation] g., mixture models, normalizing ﬂows, or autoregressive densities) can approximate P Z mix ; notably, the bound can be made tight if P Z mix ∈ Q .

Original: Also, one key feature of this upper bound is that the cross entropy term can be evaluated using only samples from P X j and the transformations T j ,
Revised: Also, one key feature of this upper bound is that the cross entropy term can be evaluated using only samples from P X j and the transformations T j ,

Original: i.e., H c
Revised: i.e., H c

Original: ( P Z mix , Q ) =
Revised: ( P Z mix , Q ) =

Original: (cid:80) j w j E P Xj [ − log Q
Revised: (cid:80) j w j E P Xj [ − log Q

Original: ( T j ( x j ))] .
Revised: ( T j ( x j ))] .

Original: However, we still cannot evaluate the other entropy terms H( P Z j ) since we do not know the densities of P Z j (or P X j ).
Revised: However, we still cannot evaluate the other entropy terms H( P Z j ) since we do not know the densities of P Z j (or P X j ).

Original: Thus, we leverage the fact that the T j functions are invertible to deﬁne an entropy change of variables.
Revised: Thus, we leverage the fact that the T j functions are invertible to deﬁne an entropy change of variables.

Original: Lemma 2 (Entropy Change of Variables) .
Revised: Lemma 2 (Entropy Change of Variables) .

Original: Let X ∼ P X and Z (cid:44) T ( X ) ∼ P Z , where T is an invertible transformation.
Revised: Let X ∼ P X and Z (cid:44) T ( X ) ∼ P Z , where T is an invertible transformation.

Original: The entropy of Z can be decomposed as follows:
Revised: The entropy of Z can be decomposed as follows:

Original: [Equation]
Revised: [Equation]

Original: where | J T ( x ) | is the determinant of the Jacobian of T .
Revised: where | J T ( x ) | is the determinant of the Jacobian of T .

Original: The key insight from this lemma is that H( P X ) is a constant with respect to T and can thus be ignored when optimizing
Revised: The key insight from this lemma is that H( P X ) is a constant with respect to T and can thus be ignored when optimizing

Original: T , while E P X
Revised: T , while E P X

Original: [log | J T ( x ) | ] can be approximated using only samples from P X .
Revised: [log | J T ( x ) | ] can be approximated using only samples from P X .

Original: Combining Theorem 1 and Lemma 2, we can arrive at our ﬁnal objective function which is equivalent to minimizing an upper bound on the GJSD:
Revised: Combining Theorem 1 and Lemma 2, we can arrive at our ﬁnal objective function which is equivalent to minimizing an upper bound on the GJSD:

Original: [Equation]
Revised: [Equation]

Original: where the last term − (cid:80) j w j H( P X j ) is constant with respect to T j functions so they can be ignored.
Revised: where the last term − (cid:80) j w j H( P X j ) is constant with respect to T j functions so they can be ignored.

Original: We formally deﬁne this loss function as follows.
Revised: We formally deﬁne this loss function as follows.

Original: Deﬁnition 2 (Alignment Upper Bound Loss) .
Revised: Deﬁnition 2 (Alignment Upper Bound Loss) .

Original: Given k continuous distributions { P X j } kj =1 , a class of continuous distributions Q , and a probability weight vector w , the alignment upper bound loss is deﬁned as follows:
Revised: Given k continuous distributions { P X j } kj =1 , a class of continuous distributions Q , and a probability weight vector w , the alignment upper bound loss is deﬁned as follows:

Original: [Equation]
Revised: [Equation]

Original: where T j are invertible and | J T j ( x ) | is the absolute value of the Jacobian determinant.
Revised: where T j are invertible and | J T j ( x ) | is the absolute value of the Jacobian determinant.

Original: Notice that this alignment loss can be seen as learning the best base distribution given ﬁxed ﬂow models T j .
Revised: Notice that this alignment loss can be seen as learning the best base distribution given ﬁxed ﬂow models T j .

Original: We now consider the theoretical optimum if we optimize over all invertible functions.
Revised: We now consider the theoretical optimum if we optimize over all invertible functions.

Original: Theorem 3 (Alignment at Global Minimum of L AUB ) .
Revised: Theorem 3 (Alignment at Global Minimum of L AUB ) .

Original: If
Revised: If

Original: L AUB is minimized over the class of all invertible functions, a global minimum of L AUB implies that the latent distributions are aligned, i.e., P T j ( X j ) =
Revised: L AUB is minimized over the class of all invertible functions, a global minimum of L AUB implies that the latent distributions are aligned, i.e., P T j ( X j ) =

Original: P T j (cid:48)
Revised: P T j (cid:48)

Original: ( X j (cid:48) ) for all j (cid:54) =
Revised: ( X j (cid:48) ) for all j (cid:54) =

Original: j (cid:48) .
Revised: j (cid:48) .

Original: Notably, this result holds regardless of Q .
Revised: Notably, this result holds regardless of Q .

Original: Informally, this can be proved by showing that the problem decouples into separate normalizing ﬂow losses where Q is the base distribution and the optimum is achieved only if P T j ( X j ) =
Revised: Informally, this can be proved by showing that the problem decouples into separate normalizing ﬂow losses where Q is the base distribution and the optimum is achieved only if P T j ( X j ) =

Original: Q for all T j (formal proof in the appendix).
Revised: Q for all T j (formal proof in the appendix).

Original: This alignment of the latent distributions also implies the translation between any of the observed component distributions.
Revised: This alignment of the latent distributions also implies the translation between any of the observed component distributions.

Original: The proof follows directly from Theorem 3 and the change of variables formula.
Revised: The proof follows directly from Theorem 3 and the change of variables formula.

Original: Corollary 4 (Translation at Global Minimum of L AUB ) .
Revised: Corollary 4 (Translation at Global Minimum of L AUB ) .

Original: Similar to Theorem 3, a global minimum of L AUB implies translation between any component distributions using the inverses of T j , i.e., P T − 1 j (cid:48)
Revised: Similar to Theorem 3, a global minimum of L AUB implies translation between any component distributions using the inverses of T j , i.e., P T − 1 j (cid:48)

Original: ( T j ( X j ))
Revised: ( T j ( X j ))

Original: =
Revised: =

Original: P X j (cid:48) for all j (cid:54) =
Revised: P X j (cid:48) for all j (cid:54) =

Original: j (cid:48) .
Revised: j (cid:48) .

Original: 2.2. Regularization via Transportation Cost
Revised: 2.2. Regularization via Transportation Cost

Original: While the alignment objective is the most challenging part of UDA, we argue that regularization is also critical for practical and stable alignment (or translation) between datasets because there are many optimal alignment solutions—even inﬁnitely many in most cases (see appendix for two examples).
Revised: While the alignment objective is the most challenging part of UDA, we argue that regularization is also critical for practical and stable alignment (or translation) between datasets because there are many optimal alignment solutions—even inﬁnitely many in most cases (see appendix for two examples).

Original: We alleviate this issue by adding expected transportation cost (usually squared Euclidean distance) as a regularization to our objective inspired by optimal transport (OT) concepts.
Revised: We alleviate this issue by adding expected transportation cost (usually squared Euclidean distance) as a regularization to our objective inspired by optimal transport (OT) concepts.

Original: Deﬁnition 3 (Regularized Alignment Upper Bound Loss) .
Revised: Deﬁnition 3 (Regularized Alignment Upper Bound Loss) .

Original: Given similar setup as in Def. 2 and a transportation cost function c ( a, b ) ≥ 0 for transporting a point from a to b , the regularized alignment upper bound loss is deﬁned as:
Revised: Given similar setup as in Def. 2 and a transportation cost function c ( a, b ) ≥ 0 for transporting a point from a to b , the regularized alignment upper bound loss is deﬁned as:

Original: [Equation]
Revised: [Equation]

Original: 2.3. Relationship to Prior Works
Revised: 2.3. Relationship to Prior Works

Original: AlignFlow is special case without adversarial terms
Revised: AlignFlow is special case without adversarial terms

Original: AlignFlow (Grover et al., 2020) without adversarial loss terms is a special case of our method for two distributions where the density model class Q only contains the standard normal distribution (i.e., a singleton class) and no regularization is used (i.e., λ = 0 ).
Revised: AlignFlow (Grover et al., 2020) without adversarial loss terms is a special case of our method for two distributions where the density model class Q only contains the standard normal distribution (i.e., a singleton class) and no regularization is used (i.e., λ = 0 ).

Original: Thus, AlignFlow can be viewed as initially optimizing a poor upper bound on JSD; however, the JSD bound becomes tighter as training progresses because the latent distributions independently move towards the same normal distribution.
Revised: Thus, AlignFlow can be viewed as initially optimizing a poor upper bound on JSD; however, the JSD bound becomes tighter as training progresses because the latent distributions independently move towards the same normal distribution.

Original: LRMF is special case with only one transformation
Revised: LRMF is special case with only one transformation

Original: Log-likelihood ratio minimizing ﬂows (LRMF) (Usman et al., 2020) is also a special case of our method for only two distributions, where one transformation is ﬁxed at the identity (i.e., T 2 = Id) and no regularization is applied (i.e., λ = 0 ).
Revised: Log-likelihood ratio minimizing ﬂows (LRMF) (Usman et al., 2020) is also a special case of our method for only two distributions, where one transformation is ﬁxed at the identity (i.e., T 2 = Id) and no regularization is applied (i.e., λ = 0 ).

Original: While the ﬁnal practical LRMF objective is a special case of ours, the theory is developed from a differentbut complementary perspective. The LRMF metric developed requires an assumption about a given density model
Revised: While the ﬁnal practical LRMF objective is a special case of ours, the theory is developed from a different

Original: [Figure]
Revised: [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: are the translated images.
Revised: but complementary perspective.

Original:  class, which enables a zero point (or absolute value) of the metric to be estimated but requires ﬁtting extra domain density models.
Revised: The LRMF metric developed requires an assumption about a given density model class, which enables a zero point (or absolute value) of the metric to be estimated but requires ﬁtting extra domain density models.

Original: Usman et al.
Revised: Usman et al.

Original: (2020) also do not uncover the connection of the objective as an upper bound on JSD regardless of the density model class.
Revised: (2020) also do not uncover the connection of the objective as an upper bound on JSD regardless of the density model class.

Original: Additionally, to ensure alignment, LRMF requires that the density model class includes the true target distribution because only one invertible transform is used, while our approach can theoretically align even if the shared density model class is weak (see
Revised: Additionally, to ensure alignment, LRMF requires that the density model class includes the true target distribution because only one invertible transform is used, while our approach can theoretically align even if the shared density model class is weak (see

Original: Theorem 3 and our simulated experiments).
Revised: Theorem 3 and our simulated experiments).

Original: Cooperative versus Adversarial Networks Analogous to the generator G and the discriminator D in adversarial learning, our framework has two main networks, T j and Q z .
Revised: Cooperative versus Adversarial Networks Analogous to the generator G and the discriminator D in adversarial learning, our framework has two main networks, T j and Q z .

Original: We can use any invertible function for T j (e.g., coupling-based ﬂows (Dinh et al., 2017), neural ODE ﬂows (Grathwohl et al., 2018), or residual ﬂows (Chen et al., 2019)) and any (approximate) density models for Q z (e.g., kernel densities (in low dimensions), mixture models, autoregressive densities (Salimans et al., 2017), normalizing ﬂows (Kingma and Dhariwal, 2018), or even VAEs (Kingma and Welling, 2019)).
Revised: We can use any invertible function for T j (e.g., coupling-based ﬂows (Dinh et al., 2017), neural ODE ﬂows (Grathwohl et al., 2018), or residual ﬂows (Chen et al., 2019)) and any (approximate) density models for Q z (e.g., kernel densities (in low dimensions), mixture models, autoregressive densities (Salimans et al., 2017), normalizing ﬂows (Kingma and Dhariwal, 2018), or even VAEs (Kingma and Welling, 2019)).

Original: Thus, our framework has similar modularity compared to adversarial approaches.
Revised: Thus, our framework has similar modularity compared to adversarial approaches.

Original: In contrast, we have a min-min, i.e., cooperative, optimization problem, but our transformations must be invertible and the auxiliary density model Q z may be more difﬁcult to train than the auxiliary discriminator D .
Revised: In contrast, we have a min-min, i.e., cooperative, optimization problem, but our transformations must be invertible and the auxiliary density model Q z may be more difﬁcult to train than the auxiliary discriminator D .

Original: We expect these limitations to be alleviated as new invertible models and density models are continually being developed.
Revised: We expect these limitations to be alleviated as new invertible models and density models are continually being developed.

Original: 3. Experiments and Conclusion
Revised: 3. Experiments and Conclusion

Original: We ﬁrst demonstrate the differences of our approach to
Revised: We ﬁrst demonstrate the differences of our approach to

Original: LRMF and AlignFlow in Fig.
Revised: LRMF and AlignFlow in Fig.

Original: 1 and the importance of regularization in Fig.
Revised: 1 and the importance of regularization in Fig.

Original: 2 (see appendix for details).

Original: We also demonstrate the feasibility of our approach for highdimensional datasets in some preliminary experiments shown in Fig.
Revised: We also demonstrate the feasibility of our approach for high-dimensional datasets in some preliminary experiments shown in Fig.

Original: 3 (see appendix for details).
Revised: Please see appendix for details for experiments.

Original: Yet, scaling up our framework in practice is still a fundamental challenge, and our approach inherits some weaknesses of JSD, which may not give useful gradient information in certain cases.
Revised: Yet, scaling up our framework in practice is still a fundamental challenge, and our approach inherits some weaknesses of JSD, which may not give useful gradient information in certain cases.

Original: Thus, while these experiments and our theoretical work build a uniﬁed foundation for cooperative alignment learning, our work also open up new theoretical and practical questions.
Revised: Thus, while these experiments and our theoretical work build a uniﬁed foundation for cooperative alignment learning, our work also open up new theoretical and practical questions.

Revised: Acknowledgement

Revised: The authors acknowledge support from the Army Research

Revised: Lab through Contract number W911NF-2020-221.

Original: A. Proofs
Revised: A. Proofs

Original: Proof of Equivalence in Def. 1.
Revised: Proof of Equivalence in Def. 1.

Original: While the proof of the equivalence is well-known, we reproduce here for completeness.
Revised: While the proof of the equivalence is well-known [Equation] we reproduce here for completeness.

Original: As a reminder, the KL divergence is deﬁned as:
Revised: As a reminder [Equation] the KL divergence is deﬁned as:

Original: [Equation]
Revised: [Equation]

Original: 11) where H c ( · , · ) denotes the cross entropy and H( · ) denotes entropy.
Revised: where H c ( · , · ) denotes the cross entropy and H( · ) denotes entropy.

Original: Given this, we can now easily derive the equivalence:[Equation]
Revised: Given this [Equation] we can now easily derive the equivalence:

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Proof of Lemma 2.
Revised: Proof of Lemma 2.

Original: First
Revised: First, we note the following fact from the standard change of variables formula:

Original: [Equation]
Revised: [Equation]

Original: we note thefollowing fact from the standard change of variables formula:
Revised: We can now derive our result using the change of variables for expectations (i

Original: [Equation]
Revised: [Equation]

Original: We can now derive our result using the change of variables for expectations (i.e., LOTUS) and the probability change of variables from above:
Revised: e., LOTUS) and the probability change of variables from above:

Original: [Equation]
Revised: [Equation]

Original: Proof of Theorem 3.
Revised: Proof of Theorem 3.

Original: Given any ﬁxed Q
Revised: Given any ﬁxed Q

Original: [Equation]
Revised: [Equation]

Original: minimizing L AUB decouples into minimizing separate normalizing ﬂow losses where Q is the base distribution
Revised: minimizing L AUB decouples into minimizing separate normalizing ﬂow losses where Q is the base distribution.

Original: [Equation]

Original: For each normalizing ﬂow
Revised: For each normalizing ﬂow

Original: [Equation]
Revised: [Equation]

Original: there exists an invertible T j such that T j ( X j ) ∼ Q
Revised: there exists an invertible T j such that T j ( X j ) ∼ Q

Original: [Equation]
Revised: [Equation]

Original: and this achieves the minimum value of L AUB .
Revised: and this achieves the minimum value of L AUB .

Original: More formally,
Revised: More formally,

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Original: Given that KL( P, Q ) ≥ 0 and equal to 0 if and only if P = Q , the global minimum is achieved only if P T j ( X j )
Revised: Given that KL( P, Q ) ≥ 0 and equal to 0 if and only if P = Q , the global minimum is achieved only if P T j ( X j )

Original: =
Revised: =

Original: Q, ∀ j and there exist such invertible functions (e.g., the optimal Monge map between P X j and Q for squared Euclidean transportation cost (Peyr´e and Cuturi, 2019)).
Revised: Q, ∀ j and there exist such invertible functions (e.g., the optimal Monge map between P X j and Q for squared Euclidean transportation cost (Peyr´e and Cuturi, 2019)).

Original: Additionally, the optimal value is (cid:80) j w j H( P X j ) , which is constant with respect to the T j transformations.
Revised: Additionally, the optimal value is (cid:80) j w j H( P X j ) , which is constant with respect to the T j transformations.

Original: B. Examples of Non-Unique Alignment Solutions B.1. Gaussian Example
Revised: B. Examples of Non-Unique Alignment Solutions B.1. Gaussian Example

Original: Suppose the component distributions are normal distributions, i.e., X 1 ∼ N ( µ 1 , I ) and X 2 ∼ N ( µ 2 , I ) , and for even greater simplicity, we assume T 2 is the identity, i.e., T 2 ( x )
Revised: Suppose the component distributions are normal distributions, i.e., X 1 ∼ N ( µ 1 , I ) and X 2 ∼ N ( µ 2 , I ) , and for even greater simplicity, we assume T 2 is the identity, i.e., T 2 ( x )

Original: = x .
Revised: = x .

Original: Then, a global optimal solution could be T 1 ( x ) =
Revised: Then, a global optimal solution could be T 1 ( x ) =

Original: U ( x − µ 1 + µ 2 ) for any orthogonal matrix U , i.e., there are inﬁnitely many invertible functions that align the distributions.
Revised: U ( x − µ 1 + µ 2 ) for any orthogonal matrix U , i.e., there are inﬁnitely many invertible functions that align the distributions.

Original: Note that this lack of unique solutions is not restricted to orthogonal rotations (see appendix for a more complex example).
Revised: Note that this lack of unique solutions is not restricted to orthogonal rotations (see appendix for a more complex example).

Original: B.2. Complex Example
Revised: B.2. Complex Example

Original: Consider the 1D case where Q only contains the uniform distribution.
Revised: Consider the 1D case where Q only contains the uniform distribution.

Original: Thus, T 1 and T 2 must map their distributions to the uniform distribution for alignment.
Revised: Thus, T 1 and T 2 must map their distributions to the uniform distribution for alignment.

Original: One solution would be that T 1 = F 1 and T 2 = F 2 where F 1 and F 2 are the CDFs of
Revised: One solution would be that T 1 = F 1 and T 2 = F 2 where F 1 and F 2 are the CDFs of

Original: P X 1 and P X 2 .
Revised: P X 1 and P X 2 .

Original: Yet, there are inﬁnitely many other possible solutions.
Revised: Yet, there are inﬁnitely many other possible solutions.

Original: Consider an invertible function that subdivides the unit interval into an arbitrarily large number of equal length intervals and then shufﬂes these intervals with a ﬁxed arbitrary permutation.
Revised: Consider an invertible function that subdivides the unit interval into an arbitrarily large number of equal length intervals and then shufﬂes these intervals with a ﬁxed arbitrary permutation.

Original: More formally, we could deﬁne this as:
Revised: More formally, we could deﬁne this as:

Original: [Equation]
Revised: [Equation]

Original: where π ( · ) is a permutation of the integers 1 to m .
Revised: where π ( · ) is a permutation of the integers 1 to m .

Original: Given this, then other optimal solutions could be T 1 = S m,π ◦ F 1 and
Revised: Given this, then other optimal solutions could be T 1 = S m,π ◦ F 1 and

Original: T 2 = F 2 for any m > 1 and any permutation π .
Revised: T 2 = F 2 for any m > 1 and any permutation π .

Original: This idea could be generalized to higher dimensions as well by mapping to the multivariate uniform distribution and subdividing the unit hypercube similarly.
Revised: This idea could be generalized to higher dimensions as well by mapping to the multivariate uniform distribution and subdividing the unit hypercube similarly.

Original: C. 2D dataset comparison with related works C.1. Single T vs. Double T ’s (LRMF vs. Ours)
Revised: C. 2D dataset comparison with related works C.1. Single T vs. Double T ’s (LRMF vs. Ours)

Original: We ﬁrst compare our method with LRMF (Usman et al., 2020) method.
Revised: We ﬁrst compare our method with LRMF (Usman et al., 2020) method.

Original: We construct the experiment to have the task:
Revised: We construct the experiment to have the task:

Original: Transform between the two half-circled distributions X 1 and X 2 in the moons dataset.
Revised: Transform between the two half-circled distributions X 1 and X 2 in the moons dataset.

Original: In this example, we made two models, one with LRMF setup and one with our RAUB setup.
Revised: In this example, we made two models, one with LRMF setup and one with our RAUB setup.

Original: As illustrated in Figure 1, the LRMF method fails to transform between X 1 and X 2 .
Revised: As illustrated in Figure 1, the LRMF method fails to transform between X 1 and X 2 .

Original: Even though Q can model well enough for T 1 ( X 1 ) , Q can only model the mean and variance of T 2 ( X 2 ) which is obviously not informative enough.
Revised: Even though Q can model well enough for T 1 ( X 1 ) , Q can only model the mean and variance of T 2 ( X 2 ) which is obviously not informative enough.

Original: Therefore, the LRMF fails to transform between two datasets.
Revised: Therefore, the LRMF fails to transform between two datasets.

Original: While in the RAUB setup, both T 1 ( X 1 ) and T 2 ( X 2 ) are modeled to the same distribution while Q can model well enough.
Revised: While in the RAUB setup, both T 1 ( X 1 ) and T 2 ( X 2 ) are modeled to the same distribution while Q can model well enough.

Original: And the resulting inverted version of X 1 and X 2 shows valid transformation.
Revised: And the resulting inverted version of X 1 and X 2 shows valid transformation.

Original: Therefore, the performance of the LRMF model is limited by the power of the density model Q which means if Q fails to model one of the transformed data distribution well enough, data alignment cannot be achieved with high performances.
Revised: Therefore, the performance of the LRMF model is limited by the power of the density model Q which means if Q fails to model one of the transformed data distribution well enough, data alignment cannot be achieved with high performances.

Original: C.2. Simple Fixed Q vs. Learnable Q (AlignFlow vs. Ours)
Revised: C.2. Simple Fixed Q vs. Learnable Q (AlignFlow vs. Ours)

Original: Next we compare our method with AlignFlow(Grover et al., 2020; Hu et al., 2018) setup.
Revised: Next we compare our method with AlignFlow(Grover et al., 2020; Hu et al., 2018) setup.

Original: We construct the experiment to have the task: Transform between the two random patterns X 1 and X 2 in the randomly generated datasets.
Revised: We construct the experiment to have the task: Transform between the two random patterns X 1 and X 2 in the randomly generated datasets.

Original: Again, we made two models with AlignFlow and our RAUB setups respectively.
Revised: Again, we made two models with AlignFlow and our RAUB setups respectively.

Original: As illustrated in Figure 1, the AlignFlow method fails to transform between X 1 and X 2 , because the transformed dataset T 1 ( X 1 ) and T 2 ( X 2 ) failed to reach the normal distribution
Revised: As illustrated in Figure 1, the AlignFlow method fails to transform between X 1 and X 2 , because the transformed dataset T 1 ( X 1 ) and T 2 ( X 2 ) failed to reach the normal distribution

Original: Q .
Revised: Q .

Original: While in the RAUB setup, the density model Q is learned to help ﬁt the transformed distributions T 1 ( X 1 ) and T 2 ( X 2 ) , which allows them to be aligned with each other.
Revised: While in the RAUB setup, the density model Q is learned to help ﬁt the transformed distributions T 1 ( X 1 ) and T 2 ( X 2 ) , which allows them to be aligned with each other.

Original: Therefore, the performance of the AlignFlow model is limited by the performances of the invertible functions.
Revised: Therefore, the performance of the AlignFlow model is limited by the performances of the invertible functions.

Original: C.3. Regularized vs. Un-regularized (Some prior works vs. Ours)
Revised: C.3. Regularized vs. Un-regularized (Some prior works vs. Ours)

Original: We ﬁnally show the importance of the regularization term.
Revised: We ﬁnally show the importance of the regularization term.

Original: We construct the experiment to have the task: Transform between two concentric circles with the same mean but slighted different radius.
Revised: We construct the experiment to have the task: Transform between two concentric circles with the same mean but slighted different radius.

Original: In this example, we make two models with RAUB setup, but one with regularization and one without.
Revised: In this example, we make two models with RAUB setup, but one with regularization and one without.

Original: As illustrated in the Figure 2, both models are able to transform between two distributions with high performances.
Revised: As illustrated in the Figure 2, both models are able to transform between two distributions with high performances.

Original: However, the transformation pattern is not natural in terms of the ’moving cost’ of each point.
Revised: However, the transformation pattern is not natural in terms of the ’moving cost’ of each point.

Original: Each pair created by the unregularized model has bigger transportation cost compared to the pairs created by the regularized model.
Revised: Each pair created by the unregularized model has bigger transportation cost compared to the pairs created by the regularized model.

Original: Therefore, we argue that by adding a transportation cost, the resulting transformation between samples would be closer to an identity transformation and therefore more stable.
Revised: Therefore, we argue that by adding a transportation cost, the resulting transformation between samples would be closer to an identity transformation and therefore more stable.

Original: Note: All implementation details on the toy dataset are available in the Appendix C.
Revised: Note: All implementation details on the toy dataset are available in the Appendix C.

Original: D. Real-world datasets
Revised: D. Real-world datasets

Original: To verify the extensibility of our fundamental idea, we also performed our experiments in real-world data.
Revised: To verify the extensibility of our fundamental idea, we also performed our experiments in real-world data.

Original: Concretely, we compare our model’s performance with the baselines in the image translation task since an evaluation on the dataset alignment are intuitive and interpretable.
Revised: Concretely, we compare our model’s performance with the baselines in the image translation task since an evaluation on the dataset alignment are intuitive and interpretable.

Original: D.1. Experiment Details
Revised: D.1. Experiment Details

Original: We use MNIST and USPS dataset in our experiments.
Revised: We use MNIST and USPS dataset in our experiments.

Original: Both two datasets are composed of hand-written digits withclasses.
Revised: Both two datasets are composed of hand-written digits withclasses.

Original: Speciﬁcally, for the simplest setting, we do our experiments with zero and one classes of MNIST datasest because a translation from zero to one and vice versa requires a semantic transformation, i.e., those two numbers cannot be transformed into each other with a simple translation or rotation.
Revised: Speciﬁcally, for the simplest setting, we do our experiments with zero and one classes of MNIST datasest because a translation from zero to one and vice versa requires a semantic transformation, i.e., those two numbers cannot be transformed into each other with a simple translation or rotation.

Original: We next perform our experiments with more complicated settings by aligning two different MNIST and USPS datasets.
Revised: We next perform our experiments with more complicated settings by aligning two different MNIST and USPS datasets.

Original: To cover the high complexity of the real-world data, we exploit a CNN-based ﬂow model (Dinh et al., 2016) as our invertible functions.
Revised: To cover the high complexity of the real-world data, we exploit a CNN-based ﬂow model (Dinh et al., 2016) as our invertible functions.

Original: We also introduce the state of the art density model (Salimans et al., 2017) as our Q .
Revised: We also introduce the state of the art density model (Salimans et al., 2017) as our Q .

Original: Regarding a training procedure, as mentioned in subsection F.1, we ﬁrst pre-train our Q z model to efﬁciently train our T functions.
Revised: Regarding a training procedure, as mentioned in subsection F.1, we ﬁrst pre-train our Q z model to efﬁciently train our T functions.

Original: We then set our frameworks to gradually transfer knowledge from Q to Ts by introducing β to our loss function, as elaborated in subsection F.2.
Revised: We then set our frameworks to gradually transfer knowledge from Q to Ts by introducing β to our loss function, as elaborated in subsection F.2.

Original: Learning rate is empirically set to be 0.002 and scheduled to be exponentially decreased after training 10 epoch with a decaying factor of 0.95.
Revised: Learning rate is empirically set to be 0.002 and scheduled to be exponentially decreased after training 10 epoch with a decaying factor of 0.95.

Original: D.2. Experiment Results
Revised: D.2. Experiment Results

Original: Fig.
Revised: Fig.

Original: 3 shows the results of the two real world datasets.
Revised: 3 shows the results of the two real world datasets.

Original: The left experiment in the ﬁgure shows that the upper bound on the JSD is effectively minimized since the translations between MNIST and USPS are decently working.
Revised: The left experiment in the ﬁgure shows that the upper bound on the JSD is effectively minimized since the translations between MNIST and USPS are decently working.

Original: This implies that common structures of the different domains, e.g., digit information are properly mapped into the shared representations
Revised: This implies that common structures of the different domains, e.g., digit information are properly mapped into the shared representations

Original: T 1 ( X 1 ) and T 2 ( X 2 ) while distinctive characteristics of the domains, e.g., a bigger and a thicker pattern of USPS can be transformed via T − 1and T − 1 2 .
Revised: T 1 ( X 1 ) and T 2 ( X 2 ) while distinctive characteristics of the domains, e.g., a bigger and a thicker pattern of USPS can be transformed via T − 1and T − 1 2 .

Original: We believe this demonstrates that the tight upper bound (by Q ) of the JSD effectively successively forms the indistinguishable representations.
Revised: We believe this demonstrates that the tight upper bound (by Q ) of the JSD effectively successively forms the indistinguishable representations.

Original: On the other hand, our proposed idea has some limitations to tackle in the realworld dataset.
Revised: On the other hand, our proposed idea has some limitations to tackle in the realworld dataset.

Original: First of all, as shown in the simpler 0 ⇔ 1 experiment in Fig.
Revised: First of all, as shown in the simpler 0 ⇔ 1 experiment in Fig.

Original: 3, our complex T with the simple Q yields the overﬁtting.
Revised: 3, our complex T with the simple Q yields the overﬁtting.

Original: Second, β in the realworld experiments has to be carefully set in training our frameworks.
Revised: Second, β in the realworld experiments has to be carefully set in training our frameworks.

Original: To elaborate, once the gradually increasing β reaches a speciﬁc value, following the fractional distributions strategy in subsection F.2, it starts to make the model performance worse.
Revised: To elaborate, once the gradually increasing β reaches a speciﬁc value, following the fractional distributions strategy in subsection F.2, it starts to make the model performance worse.

Original: Third, model performance with the higher resolution, e.g., 256 × 256 needs to be conducted to convincingly verify the performance in realworld dataset.
Revised: Third, model performance with the higher resolution, e.g., 256 × 256 needs to be conducted to convincingly verify the performance in realworld dataset.

Original: We think these limitations have to be explored to better understand the behavior of our frameworks in realworld dataset.
Revised: We think these limitations have to be explored to better understand the behavior of our frameworks in realworld dataset.

Original: E. Detailed Parameters Used in 2D Dataset Experiment E.1. LRMF vs. Ours Experiment
Revised: E. Detailed Parameters Used in 2D Dataset Experiment E.1. LRMF vs. Ours Experiment

Original: • T for LRMF setup: T 1 : 8 channel-wise mask for Real-NVP model with s and t derived from 64 hidden channels of fully connected networks.
Revised: • T for LRMF setup: T 1 : 8 channel-wise mask for Real-NVP model with s and t derived from 64 hidden channels of fully connected networks.

Original: T 2 : Identity function.
Revised: T 2 : Identity function.

Original: • T for RAUB setup: T 1 and T 1 : 8 channel-wise mask for Real-NVP model with s and t derived from 64 hidden channels of fully connected networks.
Revised: • T for RAUB setup: T 1 and T 1 : 8 channel-wise mask for Real-NVP model with s and t derived from 64 hidden channels of fully connected networks.

Original: Regularization coefﬁcient λ =• Q for both: A single Gaussian distribution with trainable mean and trainable variances.
Revised: Regularization coefﬁcient λ =• Q for both: A single Gaussian distribution with trainable mean and trainable variances.

Original: E.2.
Revised: E.2.

Original: Alignﬂow vs
Revised: Alignﬂow vs. Ours

Original: Ours Experiment • T for both: 2 channel-wise mask for RealNVP model with s and t derived from 8 hidden channels of fully connected networks.
Revised: Experiment • T for both: 2 channel-wise mask for RealNVP model with s and t derived from 8 hidden channels of fully connected networks.

Original: • Q for Alignﬂow setup: A single ﬁxed normal distribution.
Revised: • Q for Alignﬂow setup: A single ﬁxed normal distribution.

Original: • Q for RAUB setup: A learnable mixture of Gaussian with 3 components.
Revised: • Q for RAUB setup: A learnable mixture of Gaussian with 3 components.

Original: Regularization coefﬁcient λ =E.3.
Revised: Regularization coefﬁcient λ =E

Original: Regularized vs. Unregularized Experiment 1.
Revised: Regularized vs Unregularized Experiment 1.

Original: T for both: 8 channel-wise mask for RealNVP model with s and t derived from 64 hidden channels of fully connected networks.
Revised: T for both: 8 channel-wise mask for RealNVP model with s and t derived from 64 hidden channels of fully connected networks.

Original: Q for both: A learnable mixture of Gaussian with 2 components.
Revised: Q for both: A learnable mixture of Gaussian with 2 components.

Original: λ for unregularized Experiment : λ =4.
Revised: λ for unregularized Experiment : λ =4.

Original: λ for regularized Experiment : λ =
Revised: λ for regularized Experiment : λ =

Original: F. Algorithm F.1. Pre-trained Model
Revised: F. Algorithm F.1. Pre-trained Model

Original: One of the beneﬁts of our method over the baseline methods is that it is possible to harness the power of the pre-trained density model as our Q .
Revised: One of the beneﬁts of our method over the baseline methods is that it is possible to harness the power of the pre-trained density model as our Q .

Original: It is worth mentioning the use of the pre-trained Q enables the gap between the upperbound and the GJSD to be small, thus the better performance can be theoretically ensured compared to baselines with relatively simple Q .
Revised: It is worth mentioning the use of the pre-trained Q enables the gap between the upperbound and the GJSD to be small, thus the better performance can be theoretically ensured compared to baselines with relatively simple Q .

Original: Furthermore, it is possible to accomplish the statistical and computational efﬁciency in a training procedure.
Revised: Furthermore, it is possible to accomplish the statistical and computational efﬁciency in a training procedure.

Original: Speciﬁcally, the number of required data to train the networks T can be signiﬁcantly reduced and the training time can be shortened.
Revised: Speciﬁcally, the number of required data to train the networks T can be signiﬁcantly reduced and the training time can be shortened.

Original: Based on these, we leveraged one of the pre-trained state-of-the-art density model () as our Q throughout our experiment.
Revised: Based on these, we leveraged one of the pre-trained state-of-the-art density model () as our Q throughout our experiment.

Original: F.2. Train with fractional distributions
Revised: F.2. Train with fractional distributions

Original: Most state-of-the-art density would not have a nice convex structure across the entire domain of the images.
Revised: Most state-of-the-art density would not have a nice convex structure across the entire domain of the images.

Original: They will mostly have a nice peak structure within a small range from the true distributions but remain noisy for the rest domain.
Revised: They will mostly have a nice peak structure within a small range from the true distributions but remain noisy for the rest domain.

Original: This would cost the result to fall into local minimum easily at the very beginning and slow or even prevent the loss to a more desirable low value.
Revised: This would cost the result to fall into local minimum easily at the very beginning and slow or even prevent the loss to a more desirable low value.

Original: At the same time, this narrowed range of convexity pattern will make the transform function T much harder to learn so that the transformed distribution T ( x ) will fail to ﬁt the distribution of Q z .
Revised: At the same time, this narrowed range of convexity pattern will make the transform function T much harder to learn so that the transformed distribution T ( x ) will fail to ﬁt the distribution of Q z .

Original: This is also an even bigger problem if we are using a pre-trained model of Q z to begin with.
Revised: This is also an even bigger problem if we are using a pre-trained model of Q z to begin with.

Original: Therefore, in order to circumvent this kind of situation, we propose to use a fractional power of distributions of Q z at the ﬁrst epoch, and slowly increase the power up to 1 to met the original loss objectives.
Revised: Therefore, in order to circumvent this kind of situation, we propose to use a fractional power of distributions of Q z at the ﬁrst epoch, and slowly increase the power up to 1 to met the original loss objectives.

Original: The basic idea behind the newly introduced fractional power is to expand the variance of Q z so that the originally nice range of convexity will get expand, so that the algorithm should have a relatively loss function structure to start with.
Revised: The basic idea behind the newly introduced fractional power is to expand the variance of Q z so that the originally nice range of convexity will get expand, so that the algorithm should have a relatively loss function structure to start with.

Original: As the training goes, since T ( x ) is more close to the distribution of density model, we can slightly increase the fractional power to reduce the range of complexity.
Revised: As the training goes, since T ( x ) is more close to the distribution of density model, we can slightly increase the fractional power to reduce the range of complexity.

Original: After we have increased our power to 1, the objective is exactly the same of our original loss function.
Revised: After we have increased our power to 1, the objective is exactly the same of our original loss function.

Original: By using this kind of warm start epochs, we can have our T ( x ) to start at a relatively good range in
Revised: By using this kind of warm start epochs, we can have our T ( x ) to start at a relatively good range in

Original: Q z , which result in a more efﬁcient learning curve for T .
Revised: Q z , which result in a more efﬁcient learning curve for T .

Original: The implementation of this idea is also straightforward:
Revised: The implementation of this idea is also straightforward:

Original: By introducing the fractional power β , we have
Revised: By introducing the fractional power β , we have

Original: [Equation]
Revised: [Equation]