Original: General Invertible Transformations for Flow-based Generative Modeling
Revised: General Invertible Transformations for Flow-based Generative Modeling

Original: Abstract
Revised: Abstract

Original: In this paper, we present a new class of invertible transformations with an application to ﬂow-based generative models.
Revised: In this paper, we present a new class of invertible transformations with an application to ﬂow-based generative models.

Original: We indicate that many wellknown invertible transformations in reversible logic and reversible neural networks could be derived from our proposition.
Revised: We indicate that many wellknown invertible transformations in reversible logic and reversible neural networks could be derived from our proposition.

Original: Next, we propose two new coupling layers that are important building blocks of ﬂow-based generative models.
Revised: Next, we propose two new coupling layers that are important building blocks of ﬂow-based generative models.

Original: In the experiments on digit data, we present how these new coupling layers could be used in Integer Discrete Flows (IDF), and that they achieve better results than standard coupling layers used in IDF and RealNVP.
Revised: In the experiments on digit data, we present how these new coupling layers could be used in Integer Discrete Flows (IDF), and that they achieve better results than standard coupling layers used in IDF and RealNVP.

Original: 1. Introduction
Revised: 1. Introduction

Original: Notation Let us consider a D -dimensional space x ∈ X , e.g., X = { 0 , 1 } D , X
Revised: Notation Let us consider a D -dimensional space x ∈ X , e.g., X = { 0 , 1 } D , X

Original: =
Revised: =

Original: Z D or X =
Revised: Z D or X =

Original: R D .
Revised: R D .

Original: We deﬁne a binary invertible operator ◦ :
Revised: We deﬁne a binary invertible operator ◦ :

Original: X × X → X .
Revised: X × X → X .

Original: The inverse operation to ◦ is denoted by • .
Revised: The inverse operation to ◦ is denoted by • .

Original: For instance, for the addition: ◦ ≡ + and • ≡ − , and for the XOR operator: ◦ ≡ ⊕ and • ≡ ⊕ .
Revised: For instance, for the addition: ◦ ≡ + and • ≡ − , and for the XOR operator: ◦ ≡ ⊕ and • ≡ ⊕ .

Original: Further, we use the following notation: X i : j is a subset of X corresponding to variables from the i -th dimension to the j -th dimension, x i : j , we assume that X 1:0 = ∅ and X n +1: n = ∅
Revised: Further, we use the following notation: X i : j is a subset of X corresponding to variables from the i -th dimension to the j -th dimension, x i : j , we assume that X 1:0 = ∅ and X n +1: n = ∅

Original: Invertible transformations In reversible computing (Toffoli, 1980; Fredkin & Toffoli, 1982), invertible logic gates allow inverting logic operations in order to potentially decrease energy consumption of computation (Bennett, 2003).
Revised: Invertible transformations In reversible computing (Toffoli, 1980; Fredkin & Toffoli, 1982), invertible logic gates allow inverting logic operations in order to potentially decrease energy consumption of computation (Bennett, 2003).

Original: Typical invertible logic gates are:
Revised: Typical invertible logic gates are:

Original: − the
Revised: − the

Original: Feynman
Revised: Feynman

Original: gate
Revised: gate

Original: For x ∈ { 0 , 1 }
Revised: For x ∈ { 0 , 1 }

Original: , the gate is deﬁned as follows (Feynman, 1986):
Revised: , the gate is deﬁned as follows (Feynman, 1986):

Original: [Equation]
Revised: [Equation]

Original: − the Toffoli gate : For x ∈ { 0 , 1 } 3 , the gate is deﬁned as follows (Toffoli, 1980):
Revised: − the Toffoli gate : For x ∈ { 0 , 1 } 3 , the gate is deﬁned as follows (Toffoli, 1980):

Original: [Equation]
Revised: [Equation]

Original: − the Fredkin gate : For x ∈ { 0 , 1 } 3 , the gate is deﬁned as follows (Fredkin & Toffoli, 1982):
Revised: − the Fredkin gate : For x ∈ { 0 , 1 } 3 , the gate is deﬁned as follows (Fredkin & Toffoli, 1982):

Original: [Equation]
Revised: [Equation]

Original: Invertible transformations play also a crucial role in reversible neural networks (Gomez et al., 2017;
Revised: Invertible transformations play also a crucial role in reversible neural networks (Gomez et al., 2017;

Original: Chang et al., 2018; MacKay et al., 2018).
Revised: Chang et al., 2018; MacKay et al., 2018).

Original: For instance, an invertible transformation called a coupling layer is an important building block in ﬂow-based models (Dinh et al., 2016).
Revised: For instance, an invertible transformation called a coupling layer is an important building block in ﬂow-based models (Dinh et al., 2016).

Original: It is deﬁned as follows:
Revised: It is deﬁned as follows:

Original: [Equation]
Revised: [Equation]

Original: where (cid:12) is an element-wise multiplication, NN s ( · ) and
Revised: where (cid:12) is an element-wise multiplication, NN s ( · ) and

Original: NN t ( · ) denote arbitrary neural networks, and the input is divided into two parts, x =
Revised: NN t ( · ) denote arbitrary neural networks, and the input is divided into two parts, x =

Original: [ x 1 , x 2 ] , e.g., along the channel dimension.
Revised: [ x 1 , x 2 ] , e.g., along the channel dimension.

Original: If NN s ( · ) ≡ 1 , and we stack two coupling layers with reversing the order of variables in between, then we obtain the reversible residual block (Gomez et al., 2017):
Revised: If NN s ( · ) ≡ 1 , and we stack two coupling layers with reversing the order of variables in between, then we obtain the reversible residual block (Gomez et al., 2017):

Original: [Equation]
Revised: [Equation]

Original: Recently, (Hoogeboom et al., 2019) proposed a modiﬁcation of the coupling layer for integer-valued variables:
Revised: Recently, (Hoogeboom et al., 2019) proposed a modiﬁcation of the coupling layer for integer-valued variables:

Original: [Equation]
Revised: [Equation]

Original: where (cid:98)·(cid:101) denotes the rounding operation.
Revised: where (cid:98)·(cid:101) denotes the rounding operation.

Original: In order to allow applying bakcpropagation to the rounding operation, the straight through gradient estimator is used (Hoogeboom et al., 2019).
Revised: In order to allow applying bakcpropagation to the rounding operation, the straight through gradient estimator is used (Hoogeboom et al., 2019).

Original: Flow-based generative models There are three major groups of generative models: autoregressive models (Van den Oord et al., 2016)
Revised: Flow-based generative models There are three major groups of generative models: autoregressive models (Van den Oord et al., 2016)

Original: , latent variable models(Goodfellow et al., 2014; Kingma & Welling, 2013;
Revised: , latent variable models(Goodfellow et al., 2014; Kingma & Welling, 2013;

Original: Rezende et al., 2014), and ﬂow-based models (Papamakarios et al., 2019).
Revised: Rezende et al., 2014), and ﬂow-based models (Papamakarios et al., 2019).

Original: The last approach takes advantage of the change of variables formula:
Revised: The last approach takes advantage of the change of variables formula:

Original: [Equation]
Revised: [Equation]

Original: where π ( · ) is a known distribution (a base distribution, e.g.,
Revised: where π ( · ) is a known distribution (a base distribution, e.g.,

Original: Normal distribution), f : X → X is a bijective map, and
Revised: Normal distribution), f : X → X is a bijective map, and

Original: J f ( z ) denotes the Jacobian matrix.
Revised: J f ( z ) denotes the Jacobian matrix.

Original: The main challenge of ﬂow-based generative models lies in formulating invertible transformations for which the Jacobian determinant is computationally tractable.
Revised: The main challenge of ﬂow-based generative models lies in formulating invertible transformations for which the Jacobian determinant is computationally tractable.

Original: In the simplest case, we can use volume-preserving transformations that result in J f ( z ) = 1 , e.g., linear autoregressive ﬂows (Kingma et al., 2016; Tomczak & Welling, 2017) or
Revised: In the simplest case, we can use volume-preserving transformations that result in J f ( z ) = 1 , e.g., linear autoregressive ﬂows (Kingma et al., 2016; Tomczak & Welling, 2017) or

Original: Householder ﬂows (Tomczak & Welling, 2016).
Revised: Householder ﬂows (Tomczak & Welling, 2016).

Original: However, the volume-preserving ﬂows cannot model arbitrary distributions, therefore, non-linear transformations are preferable.
Revised: However, the volume-preserving ﬂows cannot model arbitrary distributions, therefore, non-linear transformations are preferable.

Original: In (Rezende & Mohamed, 2015; Berg et al., 2018;
Revised: In (Rezende & Mohamed, 2015; Berg et al., 2018;

Original: Hoogeboom et al., 2020) a speciﬁc form of non-linear transformations was constructed so that to take advantage of the matrix determinant lemma and its generalization to efﬁciently calculate the Jacobian determinant.
Revised: Hoogeboom et al., 2020) a speciﬁc form of non-linear transformations was constructed so that to take advantage of the matrix determinant lemma and its generalization to efﬁciently calculate the Jacobian determinant.

Original: Recently, the transformation used in (Rezende & Mohamed, 2015; Berg et al., 2018) was further generalized to arbitrary contractive residual networks (Chen et al., 2019) and contractive densenets (Perugachi-Diaz et al., 2021) with the Russian roulette estimator of the Jacobian determinant.
Revised: Recently, the transformation used in (Rezende & Mohamed, 2015; Berg et al., 2018) was further generalized to arbitrary contractive residual networks (Chen et al., 2019) and contractive densenets (Perugachi-Diaz et al., 2021) with the Russian roulette estimator of the Jacobian determinant.

Original: Coupling layers constitute a different group of non-linear transformation that are used in ﬂow-based models like RealNVP (Dinh et al., 2016) and GLOW (Kingma & Dhariwal, 2018).
Revised: Coupling layers constitute a different group of non-linear transformation that are used in ﬂow-based models like RealNVP (Dinh et al., 2016) and GLOW (Kingma & Dhariwal, 2018).

Original: In the case of discrete variables, e.g., X = Z D , the change of variables takes a simpler form due to the fact that there is no change of volume for probability mass functions:
Revised: In the case of discrete variables, e.g., X = Z D , the change of variables takes a simpler form due to the fact that there is no change of volume for probability mass functions:

Original: [Equation]
Revised: [Equation]

Original: To date, coupling layers with the rounding operator (Eq.
Revised: To date, coupling layers with the rounding operator (Eq.

Original: 6) are typically used.
Revised: 6) are typically used.

Original: The resulting ﬂow-based models are called Integer Discrete Flows (IDF) (Hoogeboom et al., 2019; Berg et al., 2020) with a mixture of discretized logistic distributions (Salimans et al., 2017) as the base distribution.
Revised: The resulting ﬂow-based models are called Integer Discrete Flows (IDF) (Hoogeboom et al., 2019; Berg et al., 2020) with a mixture of discretized logistic distributions (Salimans et al., 2017) as the base distribution.

Original: 2. Our approach 2.1. General invertible transformations
Revised: 2. Our approach 2.1. General invertible transformations

Original: Our main contribution of this paper is a proposition of a new class of invertible transformations that generalize many invertible transformations in reversible computing and reversible deep learning.
Revised: Our main contribution of this paper is a proposition of a new class of invertible transformations that generalize many invertible transformations in reversible computing and reversible deep learning.

Original: Proposition 1.
Revised: Proposition 1.

Original: Let us take x , y ∈ X .
Revised: Let us take x , y ∈ X .

Original: If binary transformations ◦ and (cid:46) have inverses • and (cid:74) , respectively, and g 2 , . .
Revised: If binary transformations ◦ and (cid:46) have inverses • and (cid:74) , respectively, and g 2 , . .

Original: g D and f 1 , . .
Revised: g D and f 1 , . .

Original: , f D are arbitrary functions, where
Revised: , f D are arbitrary functions, where

Original: [Equation]
Revised: [Equation]

Original: X
Revised: i , f

Original: [Equation]
Revised: [Equation]

Original: 1 × X  i [Equation]
Revised: : X 1: i − 1 ×

Original: 1: n → X i , then the following transformation from x to y :
Revised: X i +1: n → X i , then the following transformation from x to y :

Original: [Equation]
Revised: [Equation]

Original: is invertible.
Revised: is invertible.

Original: Proof.
Revised: Proof.

Original: In order to inverse y to x we start from the last element to obtain the following:
Revised: In order to inverse y to x we start from the last element to obtain the following:

Original: [Equation]
Revised: [Equation]

Original: Then, we can proceed with next expressions in the decreasing order ( i.e. , from D − 1 to 1 ) to eventually obtain:
Revised: Then, we can proceed with next expressions in the decreasing order ( i.e. , from D − 1 to 1 ) to eventually obtain:

Original: [Equation]
Revised: [Equation]

Original: Next, we show that many widely known invertible transformations could be derived from the proposed general invertible transformation.
Revised: Next, we show that many widely known invertible transformations could be derived from the proposed general invertible transformation.

Original: First, for the space of binary variables, we present that our proposition could be used to obtain three of the most important reversible logic gates.
Revised: First, for the space of binary variables, we present that our proposition could be used to obtain three of the most important reversible logic gates.

Original: Corollary 2 (Feynman gate) .
Revised: Corollary 2 (Feynman gate) .

Original: Let us consider x ∈ { 0 , 1 } 2 , and ◦ ≡ ⊕ and (cid:46) ≡ ⊕ with • ≡ ⊕ and (cid:74) ≡ ⊕ , where ⊕ is the XOR operation.
Revised: Let us consider x ∈ { 0 , 1 } 2 , and ◦ ≡ ⊕ and (cid:46) ≡ ⊕ with • ≡ ⊕ and (cid:74) ≡ ⊕ , where ⊕ is the XOR operation.

Original: Then, taking g 2 ≡ 0 , f 1 ( x 2 ) = 0 and f 2 ( y 1 ) =
Revised: Then, taking g 2 ≡ 0 , f 1 ( x 2 ) = 0 and f 2 ( y 1 ) =

Original: y 1 results in the Feynman gate:
Revised: y 1 results in the Feynman gate:

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: The Eq. 9 follows from the idempotency of XOR.
Revised: The Eq. 9 follows from the idempotency of XOR.

Original: Corollary 3 (Toffoli gate) .
Revised: Corollary 3 (Toffoli gate) .

Original: Let us consider x ∈ { 0 , 1 } 3 , and ◦ ≡ ⊕ and (cid:46) ≡ ⊕ with • ≡ ⊕ and (cid:74) ≡ ⊕ , where ⊕ is the XOR operation.
Revised: Let us consider x ∈ { 0 , 1 } 3 , and ◦ ≡ ⊕ and (cid:46) ≡ ⊕ with • ≡ ⊕ and (cid:74) ≡ ⊕ , where ⊕ is the XOR operation.

Original: Then, taking g 2 ( y 1 ) ≡ 0 , g 3 ( y 1:2 ≡ 0 , f 1 ( x 2:3 ) ≡ 0 , f 2 ( y 1 , x 3 ) ≡ 0 and f 3 ( y 1:2 ) =
Revised: Then, taking g 2 ( y 1 ) ≡ 0 , g 3 ( y 1:2 ≡ 0 , f 1 ( x 2:3 ) ≡ 0 , f 2 ( y 1 , x 3 ) ≡ 0 and f 3 ( y 1:2 ) =

Original: y 1 ∧ yresults in the Toffoli gate:
Revised: y 1 ∧ yresults in the Toffoli gate:

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: The Eqs.
Revised: The Eqs.

Original: 10 - 12 follow from the idempotency of the XOR operator.
Revised: 10 - 12 follow from the idempotency of the XOR operator.

Original: Corollary 4 (Fredkin gate) .
Revised: Corollary 4 (Fredkin gate) .

Original: Let us consider x ∈ { 0 , 1 } 4 , and ◦ ≡ ⊕ and (cid:46) ≡ ⊕ with • ≡ ⊕ and (cid:74) ≡ ⊕ , where ⊕ is the XOR operation.
Revised: Let us consider x ∈ { 0 , 1 } 4 , and ◦ ≡ ⊕ and (cid:46) ≡ ⊕ with • ≡ ⊕ and (cid:74) ≡ ⊕ , where ⊕ is the XOR operation.

Original: Then, taking x 1 ≡ 0 , g 2 ( y 1 ) ≡ 0 , g 3 ( y 1:2 ) ≡ 0 , g 4 ( y 1:3 ) ≡ 0 , f 1 ( x 2:4 ) = x 2 ∧ ( x 3 ⊕ x 4 ), f 2 ( y 1 , x 3:4 ) ≡ 0 , f 3 ( y 1:2 , x 4 ) =
Revised: Then, taking x 1 ≡ 0 , g 2 ( y 1 ) ≡ 0 , g 3 ( y 1:2 ) ≡ 0 , g 4 ( y 1:3 ) ≡ 0 , f 1 ( x 2:4 ) = x 2 ∧ ( x 3 ⊕ x 4 ), f 2 ( y 1 , x 3:4 ) ≡ 0 , f 3 ( y 1:2 , x 4 ) =

Original: y 1 and f 4 ( y 1:3 ) ≡ yresults in the Fredkin gate:
Revised: y 1 and f 4 ( y 1:3 ) ≡ yresults in the Fredkin gate:

Original: [Equation]
Revised: [Equation]

Original: Remark 5 (On the Fredkin gate) .
Revised: Remark 5 (On the Fredkin gate) .

Original: Comparing equations 13–16 with the deﬁnition of Fredkin gate we notice that in Corollary 4 we have to introduce an additional equation to be consistent with the Proposition 1.
Revised: Comparing equations 13–16 with the deﬁnition of Fredkin gate we notice that in Corollary 4 we have to introduce an additional equation to be consistent with the Proposition 1.

Original: Moreover, we introduced a dummy variable x 1 that always equals 0 .
Revised: Moreover, we introduced a dummy variable x 1 that always equals 0 .

Original: Moreover, we observe that our proposition generalizes invertible layers in neural networks.
Revised: Moreover, we observe that our proposition generalizes invertible layers in neural networks.

Original: Corollary 6 (A coupling layer) .
Revised: Corollary 6 (A coupling layer) .

Original: Let us consider x = [ x 1 , x 2 ] (cid:62) , where X i =
Revised: Let us consider x = [ x 1 , x 2 ] (cid:62) , where X i =

Original: R D i , and ◦ ≡ + , (cid:46) ≡ (cid:12) with • ≡ − and (cid:74) ≡ (cid:11) , where (cid:12) and (cid:11) denote elementwise multiplication and division, respectively.
Revised: R D i , and ◦ ≡ + , (cid:46) ≡ (cid:12) with • ≡ − and (cid:74) ≡ (cid:11) , where (cid:12) and (cid:11) denote elementwise multiplication and division, respectively.

Original: Then, taking g 2 ( y 1 ) =
Revised: Then, taking g 2 ( y 1 ) =

Original: exp(NN s
Revised: exp(NN s

Original: ( y 1 )) , f 1 ( x 2 ) = 

Revised: ( y 1 )) , f 1 ( x 2 ) =

Original: and f 2 ( y 1 ) =
Revised: and f 2 ( y 1 ) =

Original: NN t ( y 1 ) , where NN s , NN t are neural networks, results in the coupling layer (Dinh et al., 2016):
Revised: NN t ( y 1 ) , where NN s , NN t are neural networks, results in the coupling layer (Dinh et al., 2016):

Original: [Equation]
Revised: [Equation]

Original: Corollary 7 (A reversible residual layer) .
Revised: Corollary 7 (A reversible residual layer) .

Original: Let us consider x = [ x 1 , x 2 ] (cid:62) , where X i =
Revised: Let us consider x = [ x 1 , x 2 ] (cid:62) , where X i =

Original: R D i , and ◦ ≡ + , (cid:46) ≡ (cid:12) with • ≡ − and (cid:74) ≡ (cid:11) , where (cid:12) and (cid:11) denote elementwise multiplication and division, respectively.
Revised: R D i , and ◦ ≡ + , (cid:46) ≡ (cid:12) with • ≡ − and (cid:74) ≡ (cid:11) , where (cid:12) and (cid:11) denote elementwise multiplication and division, respectively.

Original: Then, taking g 2 ( y 1 ) ≡ 1 , f 1 ( x 2 ) =
Revised: Then, taking g 2 ( y 1 ) ≡ 1 , f 1 ( x 2 ) =

Original: NN 1 ( x 2 ) and f 2 ( y 1 )
Revised: NN 1 ( x 2 ) and f 2 ( y 1 )

Original: =
Revised: =

Original: NN 2 ( y 1 ) , where NN is a neural network, results in the reversible residual layer proposed in (Gomez et al., 2017):
Revised: NN 2 ( y 1 ) , where NN is a neural network, results in the reversible residual layer proposed in (Gomez et al., 2017):

Original: [Equation]
Revised: [Equation]

Original: Remark 8 (On the reversible residual layer) .
Revised: Remark 8 (On the reversible residual layer) .

Original: According to Proposition 1, we can further generalize the reversible residual layer proposed in (Gomez et al., 2017) by taking
Revised: According to Proposition 1, we can further generalize the reversible residual layer proposed in (Gomez et al., 2017) by taking

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: [Equation]
Revised: [Equation]

Original: y 1 )) that would result in the following invertible layer:
Revised: y 1 )) that would result in the following invertible layer:

Original: [Equation]
Revised: [Equation]

Original: Interestingly, we can calculate the Jacobian of such transformation that takes the following form:
Revised: Interestingly, we can calculate the Jacobian of such transformation that takes the following form:

Original: [Equation]
Revised: [Equation]

Original: where A = I , i.e., the identity matrix.
Revised: where A = I , i.e., the identity matrix.

Original: Then, AC =
Revised: Then, AC =

Original: CA and according to Theorem 3 on determinants of block matrices in (Silvester, 2000), the logarithm of the Jacobiandeterminant equals:
Revised: CA and according to Theorem 3 on determinants of block matrices in (Silvester, 2000), the logarithm of the Jacobiandeterminant equals:

Original: [Equation]
Revised: [Equation]

Original: Corollary 9 (A reversible differential mutation) .
Revised: Corollary 9 (A reversible differential mutation) .

Original: Let us consider x 1 , x 2 , x 3 ∈ R D , γ ∈ R + , and ◦ ≡ + , (cid:46) ≡ (cid:12) with • ≡ − .
Revised: Let us consider x 1 , x 2 , x 3 ∈ R D , γ ∈ R + , and ◦ ≡ + , (cid:46) ≡ (cid:12) with • ≡ − .

Original: Then, taking g 2 ( y 1 ) ≡ 1 , g 3 ( y 1:2 ) ≡ 1 , f 1 ( x 2:3 ) = γ ( x 2 − x 3 ) , f 2 ( y 1 , x 3 )
Revised: Then, taking g 2 ( y 1 ) ≡ 1 , g 3 ( y 1:2 ) ≡ 1 , f 1 ( x 2:3 ) = γ ( x 2 − x 3 ) , f 2 ( y 1 , x 3 )

Original: =
Revised: =

Original: γ ( x 3 − y 1 ) , and f 3 ( y 1:2 ) =
Revised: γ ( x 3 − y 1 ) , and f 3 ( y 1:2 ) =

Original: γ ( y 1 − y 2 ) , results in the reversible differential mutation proposed in (Tomczak et al., 2020):
Revised: γ ( y 1 − y 2 ) , results in the reversible differential mutation proposed in (Tomczak et al., 2020):

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: 2.2. General Invertible Transformations for Integer Discrete Flows
Revised: 2.2. General Invertible Transformations for Integer Discrete Flows

Original: We propose to utilize our general invertible transformations in IDF.
Revised: We propose to utilize our general invertible transformations in IDF.

Original: For this purpose, we formulate two new coupling layers that fulﬁll Proposition 1, namely: − We divide the input into four parts, x = [ x 1 , x 2 , x 3 , x 4 ] : y 

Revised: For this purpose, we formulate two new coupling layers that fulﬁll Proposition 1, namely: − We divide the input into four parts, x = [ x 1 , x 2 , x 3 , x 4 ] : y

Original: = x 1 + (cid:98) NN t, 1 ( x 2:4 ) (cid:101)
Revised: = x 1 + (cid:98) NN t, 1 ( x 2:4 ) (cid:101)

Original: y 

Revised: y

Original: = x 2 + (cid:98) NN t, 2 ( y 1 , x 3:4 ) (cid:101)
Revised: = x 2 + (cid:98) NN t, 2 ( y 1 , x 3:4 ) (cid:101)

Original: [Equation]
Revised: [Equation]

Original: − We divide the input into eight parts, x = [ x 1 , . . .
Revised: − We divide the input into eight parts, x = [ x 1 , . . .

Original: , x 8 ] .
Revised: , x 8 ] .

Original: Then, we formulate the coupling layer analogically to (28).
Revised: Then, we formulate the coupling layer analogically to (28).

Original: Further, we use the discretized two-parameter logistic distribution.
Revised: Further, we use the discretized two-parameter logistic distribution.

Original: It could be expressed as a difference of the logistic cumulative distribution functions (Salimans et al., 2017) or analytically (Chakraborty & Chakravarty, 2016).
Revised: It could be expressed as a difference of the logistic cumulative distribution functions (Salimans et al., 2017) or analytically (Chakraborty & Chakravarty, 2016).

Original: 3. Experiments
Revised: 3. Experiments

Original: Data In the experiment, we use a toy dataset of handwritten digits available in Scikit-learn 1 that consists of 1 , 797 images.
Revised: Data In the experiment, we use a toy dataset of handwritten digits available in Scikit-learn 1 that consists of 1 , 797 images.

Original: Each image consists of 64 pixels ( 8 px × 8 px).
Revised: Each image consists of 64 pixels ( 8 px × 8 px).

Original: Models In the experiment, we compare the following models:
Revised: Models In the experiment, we compare the following models:

Original: RealNVP with uniform dequantization and the standard
Revised: RealNVP with uniform dequantization and the standard

Original: Gaussian base distribution (R EAL NVP), RealNVP with the coupling layer in Remark 8 and the standard Gaussian base distribution (R EAL NVP2), IDF with the coupling layer in (6) (R EAL NVP), IDF with the coupling layer in (28) (IDF4), IDF with the coupling layer for eight parts (IDF8).
Revised: Gaussian base distribution (R EAL NVP), RealNVP with the coupling layer in Remark 8 and the standard Gaussian base distribution (R EAL NVP2), IDF with the coupling layer in (6) (R EAL NVP), IDF with the coupling layer in (28) (IDF4), IDF with the coupling layer for eight parts (IDF8).

Original: In all models, we used the order permutation (i.e., a matrix with ones on the anti-diagonal and zeros elsewhere) after each coupling layer.
Revised: In all models, we used the order permutation (i.e., a matrix with ones on the anti-diagonal and zeros elsewhere) after each coupling layer.

Original: In order to keep a similar number of weights, we usedﬂows for IDF, 8 ﬂows for R EAL NVP, 4 ﬂows for R EALNVP2, 4 ﬂows for IDF4, and 2 ﬂows for IDF8.
Revised: In order to keep a similar number of weights, we usedﬂows for IDF, 8 ﬂows for R EAL NVP, 4 ﬂows for R EALNVP2, 4 ﬂows for IDF4, and 2 ﬂows for IDF8.

Original: All models have roughly 1 .
Revised: All models have roughly 1 .

Original: 32 M weights.
Revised: 32 M weights.

Original: For IDF, IDF4, and IDF we utilized the following neural networks for transitions ( NN t ):
Revised: For IDF, IDF4, and IDF we utilized the following neural networks for transitions ( NN t ):

Original: [Equation]
Revised: [Equation]

Original: and for R EAL NVP, we additionally used the following neural networks for scaling ( NN s ):
Revised: and for R EAL NVP, we additionally used the following neural networks for scaling ( NN s ):

Original: [Equation]
Revised: [Equation]

Original: Training & Evaluation We compare the models using the negative-log likelihood (nll).
Revised: Training & Evaluation We compare the models using the negative-log likelihood (nll).

Original: We train each model using 1 , 000 images, and the mini-batch equals 64 .
Revised: We train each model using 1 , 000 images, and the mini-batch equals 64 .

Original: Moreover, we take 350 images for validation and 447 for testing.
Revised: Moreover, we take 350 images for validation and 447 for testing.

Original: Each model is trained 5 times.
Revised: Each model is trained 5 times.

Original: During training, we use the early stopping with the patience equal 20 epochs, and the best performing model on the validation set is later evaluated on the test set.
Revised: During training, we use the early stopping with the patience equal 20 epochs, and the best performing model on the validation set is later evaluated on the test set.

Original: The Adam optimizer with the learning rate equal 0 .
Revised: The Adam optimizer with the learning rate equal 0 .

Original: 001 was used.
Revised: 001 was used.

Original: Code The experiments could be reproduced by running the code available at: https://github.com/ jmtomczak/git_flow
Revised: Code The experiments could be reproduced by running the code available at: https://github.com/ jmtomczak/git_flow

Original: Results In Figure 1, we present aggregated results for the ﬁve models.
Revised: Results In Figure 1, we present aggregated results for the ﬁve models.

Original: Moreover, in Figure 2, examples of unconditional samples are presented, together with a real sample.
Revised: Moreover, in Figure 2, examples of unconditional samples are presented, together with a real sample.

Original: First, we notice that IDF performs better than R EAL NVP and R EAL NVP2.
Revised: First, we notice that IDF performs better than R EAL NVP and R EAL NVP2.

Original: In this paper, we use fully-connected neural networks and small toy data.
Revised: In this paper, we use fully-connected neural networks and small toy data.

Original: Nevertheless, it is interesting to see that IDF could perform better than a widely used
Revised: Nevertheless, it is interesting to see that IDF could perform better than a widely used

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: continuous ﬂow-based model with dequantization.
Revised: continuous ﬂow-based model with dequantization.

Original: Moreover, it seems that the new coupling layer presented in Remark 8 is more stable in terms of ﬁnal results, however, the difference in performance is statistically insigniﬁcant.
Revised: Moreover, it seems that the new coupling layer presented in Remark 8 is more stable in terms of ﬁnal results, however, the difference in performance is statistically insigniﬁcant.

Original: Second, we observe that our proposition of invertible transformations results in improved nll.
Revised: Second, we observe that our proposition of invertible transformations results in improved nll.

Original: The proposed coupling layer with 8 partitions performed the best in terms of nll, and the coupling layer with 4 partitions also outperformed
Revised: The proposed coupling layer with 8 partitions performed the best in terms of nll, and the coupling layer with 4 partitions also outperformed

Original: IDF, R EAL NVP and R EAL NVP2.
Revised: IDF, R EAL NVP and R EAL NVP2.

Original: We want to highlight that all models have almost identical numbers of weights, thus, these results are not caused by taking larger neural networks.
Revised: We want to highlight that all models have almost identical numbers of weights, thus, these results are not caused by taking larger neural networks.

Original: The samples presented in Figure 2 are rather hard to analyze due to their small size ( 8 px × 8 px).
Revised: The samples presented in Figure 2 are rather hard to analyze due to their small size ( 8 px × 8 px).

Original: Nevertheless, we notice that all IDFs generated crisper images than R EAL NVP and R EAL NVP2.
Revised: Nevertheless, we notice that all IDFs generated crisper images than R EAL NVP and R EAL NVP2.

Original: Moreover, it seems that IDF and IDF8 seem to produce digits of higher visual quality.
Revised: Moreover, it seems that IDF and IDF8 seem to produce digits of higher visual quality.

Original: 4. Conclusion
Revised: 4. Conclusion

Original: In this paper, we proposed a new class of invertible transformations.
Revised: In this paper, we proposed a new class of invertible transformations.

Original: We showed that many well-known invertible transformations could be derived from our proposition.
Revised: We showed that many well-known invertible transformations could be derived from our proposition.

Original: Moreover, we proposed two coupling layers and presented how they could be utilized in ﬂow-based models for integer values (Integer Discrete Flows).
Revised: Moreover, we proposed two coupling layers and presented how they could be utilized in ﬂow-based models for integer values (Integer Discrete Flows).

Original: Our preliminary experiments on the digits data indicate that the new coupling layers result in better negative log-likelihood values than for IDF and
Revised: Our preliminary experiments on the digits data indicate that the new coupling layers result in better negative log-likelihood values than for IDF and

Original: R EAL NVP.
Revised: R EAL NVP.

Original: These results are promising and will be pursued in the future on more challenging datasets.
Revised: These results are promising and will be pursued in the future on more challenging datasets.