Original: i-DenseNets Anonymous Authors Anonymous Institution
Revised: Invertible DenseNets

Original: Abstract
Revised: Abstract

Original: We introduce Invertible Dense Networks (i-DenseNets), a more parameter eﬃcient alternative to Residual Flows.
Revised: We introduce Invertible Dense Networks (i-DenseNets), a more parameter eﬃcient alternative to Residual Flows.

Original: The method relies on an analysis of the Lipschitz continuity of the concatenation in DenseNets, where we enforce the invertibility of the network by satisfying the Lipschitz constraint.
Revised: The method relies on an analysis of the Lipschitz continuity of the concatenation in DenseNets, where we enforce the invertibility of the network by satisfying the Lipschitz constraint.

Original: Additionally, we extend this method by proposing a learnable concatenation, which not only improves the model performance but also indicates the importance of the concatenated representation.
Revised: Additionally, we extend this method by proposing a learnable concatenation, which not only improves the model performance but also indicates the importance of the concatenated representation.

Original: We demonstrate the performance of i-DenseNets and Residual Flows on toy, MNIST, and CIFAR10 data.
Revised: We demonstrate the performance of i-DenseNets and Residual Flows on toy, MNIST, and CIFAR10 data.

Original: Both i-DenseNets outperform Residual Flows evaluated in negative log-likelihood, on all considered datasets under an equal parameter budget.
Revised: Both i-DenseNets outperform Residual Flows evaluated in negative log-likelihood, on all considered datasets under an equal parameter budget.

Original: 1. Introduction
Revised: 1. Introduction

Original: Neural networks are frequently used in supervised learning tasks such as classiﬁcation, where models are trained to predict labels.
Revised: Neural networks are frequently used in supervised learning tasks such as classiﬁcation, where models are trained to predict labels.

Original: However, they are also used to parameterize generative models that try to estimate the true distribution of the observed data.
Revised: However, they are also used to parameterize generative models that try to estimate the true distribution of the observed data.

Original: Generative models can be used to generate realistic-looking images that are hard to separate from real ones, detection of adversarial attacks (Fetaya et al., 2019; Jacobsen et al., 2018), and for hybrid modeling (Nalisnick et al., 2019) which have the property to both classify and generate.
Revised: Generative models can be used to generate realistic-looking images that are hard to separate from real ones, detection of adversarial attacks (Fetaya et al., 2019; Jacobsen et al., 2018) and for hybrid modeling (Nalisnick et al., 2019) which have the property to both classify and generate.

Original: The generative architecture come in diﬀerent designs.
Revised: The generative architecture come in diﬀerent designs.

Original: A common approach to train generative models is using the likelihood objective.
Revised: A common approach to train generative models is using the likelihood objective.

Original: One kind of model that also uses this approach are ﬂow-based models.
Revised: One kind of model that also uses this approach are ﬂow-based models.

Original: Flow-based models consist of invertible transformations that allow them to compute the likelihood using the change of variable formula.
Revised: Flow-based models consist of invertible transformations that allow them to compute the likelihood using the change of variable formula.

Original: The main diﬀerence that determines an exact computation or approximation of a ﬂow-based model, lies in the design of the transformation layer.
Revised: The main diﬀerence that determines an exact computation or approximation of a ﬂow-based model, lies in the design of the transformation layer.

Original: The design used to make this layer invertible can consist of the exact computation of the inverse or a numerical technique.
Revised: The design used to make this layer invertible can consist of the exact computation of the inverse or a numerical technique.

Original: For example, (Dinh et al., 2016) use coupling layers that consist of functions stacked on each other to make the ﬂow invertible.
Revised: For example, (Dinh et al., 2016) use coupling layers that consist of functions stacked on each other to make the ﬂow invertible.

Original: This allows an exact computation while modeling complex convolutional neural networks that do not require the computation of the derivative.
Revised: This allows an exact computation while modeling complex convolutional neural networks that do not require the computation of the derivative.

Original: Recently, Behrmann et al.
Revised: Recently, Behrmann et al.

Original: (2018) have proposed deep-residual blocks as a transformation layer.
Revised: (2018) have proposed deep-residual blocks as a transformation layer.

Original: The deep-residual networks (ResNets) of (He et al., 2016) are known for their successes in supervised learning approaches.
Revised: The deep-residual networks (ResNets) of (He et al., 2016) are known for their successes in supervised learning approaches.

Original: In a ResNet block, each input of the block is added to the output, which forms the input for the next block.
Revised: In a ResNet block, each input of the block is added to the output, which forms the input for the next block.

Original: Since ResNets are not necessarily invertible, Behrmann et al.
Revised: Since ResNets are not necessarily invertible, Behrmann et al.

Original: (2018) enforce the Lipschitz constraint in such a manner that the network becomes invertible.
Revised: (2018) enforce the Lipschitz constraint in such a manner that the network becomes invertible.

Original: Furthermore, Chen et al.
Revised: Furthermore, Chen et al.

Original: (2019) proposed Residual Flows, an improvement of i-ResNets, that uses an unbiased estimator of the log-determinant, which results in even better performance.
Revised: (2019) proposed Residual Flows, an improvement of i-ResNets, that uses an unbiased estimator of the log-determinant, which results in even better performance.

Original: In supervised learning, an architecture that uses fewer parameters and is even more powerful than the deep-residual network, is the Densely Connected Convolution Network (DenseNet), which was ﬁrst presented in (Huang et al., 2017).
Revised: In supervised learning, an architecture that uses fewer parameters and is even more powerful than the deep-residual network, is the Densely Connected Convolution Network (DenseNet), which was ﬁrst presented in (Huang et al., 2017).

Original: The network showed to improve signiﬁcantly in recognition tasks on benchmark datasets such as CIFAR, SVHN, and
Revised: The network showed to improve signiﬁcantly in recognition tasks on benchmark datasets such as CIFAR, SVHN, and

Original: ImageNet, by using fewer computations and having fewer parameters than ResNets while performing at a similar level.
Revised: ImageNet, by using fewer computations and having fewer parameters than ResNets while performing at a similar level.

Original: In contrary to a ResNet block, a DenseNet layer consists of a concatenation of the input with the output.
Revised: In contrary to a ResNet block, a DenseNet layer consists of a concatenation of the input with the output.

Original: In this work, we introduce invertible Dense
Revised: In this work, we introduce invertible Dense

Original: Networks (i-DenseNets), and we show that we can enforce the Lipschitz constraint in a similar manner as in (Behrmann et al., 2018).
Revised: Networks (i-DenseNets), and we show that we can enforce the Lipschitz constraint in a similar manner as in (Behrmann et al., 2018).

Original: Further, we show that this model can be eﬃciently trained as a generative model and outperforms Residual Flows under an equal parameter budget.
Revised: Further, we show that this model can be eﬃciently trained as a generative model and outperforms Residual Flows under an equal parameter budget.

Original: 2. Background
Revised: 2. Background

Original: Let us consider a vector of observable variables x ∈ R d and a vector of latent variables z ∈ R d .
Revised: Let us consider a vector of observable variables x ∈ R d and a vector of latent variables z ∈ R d .

Original: We deﬁne a bijective function f : R d → R d which maps a latent variable to datapoint x = f ( z ).
Revised: We deﬁne a bijective function f : R d → R d which maps a latent variable to datapoint x = f ( z ).

Original: If f is invertible, we deﬁne its inverse as F = f − 1 .
Revised: If f is invertible, we deﬁne its inverse as F = f − 1 .

Original: Further, we use the change of variable formula to compute the likelihood of a datapoint x by:
Revised: Further, we use the change of variable formula to compute the likelihood of a datapoint x by:

Original: [Equation]
Revised: [Equation]

Original: where p Z ( z ) is a base distribution (e.g., the standard Gaussian) and J F ( x ) is the Jacobian of F at x .
Revised: where p Z ( z ) is a base distribution (e.g., the standard Gaussian) and J F ( x ) is the Jacobian of F at x .

Original: The change of variable formula allows tractable evaluation of the data and the ﬂows are trained using the maximum likelihood objective.
Revised: The change of variable formula allows tractable evaluation of the data and the ﬂows are trained using the maximum likelihood objective.

Original: Behrmann et al.
Revised: Behrmann et al.

Original: (2018) construct an invertible ResNet layer which is only constraint in Lipschitz continuity.
Revised: (2018) construct an invertible ResNet layer which is only constraint in Lipschitz continuity.

Original: A ResNet is deﬁned as: F ( x ) = x
Revised: A ResNet is deﬁned as: F ( x ) = x

Original: + g ( x ), where g is modeled by a (convolutional) neural network and F represents a ResNet layer which is in general not invertible.
Revised: + g ( x ), where g is modeled by a (convolutional) neural network and F represents a ResNet layer which is in general not invertible.

Original: However, they construct g in such way to satisfy Lip(g) < 1 by using spectral normalization of (Gouk et al., 2018;
Revised: However, they construct g in such way to satisfy Lip(g) < 1 by using spectral normalization of (Gouk et al., 2018;

Original: Miyato et al., 2018), such that:
Revised: Miyato et al., 2018), such that:

Original: [Equation]
Revised: [Equation]

Original: where || · || 2 is the (cid:96) 2 norm.
Revised: where || · || 2 is the (cid:96) 2 norm.

Original: Since the Banach ﬁxed-point theorem holds in this speciﬁc case, the ResNet layer F has a unique inverse, even though there does not need to be an analytical closed-form solution.
Revised: Since the Banach ﬁxed-point theorem holds in this speciﬁc case, the ResNet layer F has a unique inverse, even though there does not need to be an analytical closed-form solution.

Original: Further, the log-determinant can be estimated using the
Revised: Further, the log-determinant can be estimated using the

Original: Hutchinsons trace estimator (Skilling, 1989; Hutchinson, 1990), at a lower cost than to fully compute the trace of the Jacobian.
Revised: Hutchinsons trace estimator (Skilling, 1989; Hutchinson, 1990), at a lower cost than to fully compute the trace of the Jacobian.

Original: Chen et al.
Revised: Chen et al.

Original: (2019) propose Residual Flows that uses an improved method to estimate the log-determinant with an unbiased estimator.
Revised: (2019) propose Residual Flows that uses an improved method to estimate the log-determinant with an unbiased estimator.

Original: 3. Invertible Dense Networks
Revised: 3. Invertible Dense Networks

Original: We introduce i-DenseNets, an invertible model based on DenseNets parametrizations.
Revised: We introduce i-DenseNets, an invertible model based on DenseNets parametrizations.

Original: To formulate i-DenseNets, we deﬁne a DenseBlock as a function F : R d → R d with F ( x ) =
Revised: To formulate i-DenseNets, we deﬁne a DenseBlock as a function F : R d → R d with F ( x ) =

Original: x + g ( x ), where g consists of Dense Layers { h i } ni =1 that are expressed as:
Revised: x + g ( x ), where g consists of Dense Layers { h i } ni =1 that are expressed as:

Original: [Equation]
Revised: [Equation]

Original: where h n +1 represents a 1 × 1 convolution to match the output size of R d .
Revised: where h n +1 represents a 1 × 1 convolution to match the output size of R d .

Original: A layer h
Revised: A layer h

Original: i consist of two parts concatenated to each other.
Revised: i consist of two parts concatenated to each other.

Original: The upper part is a copy of the input signal.
Revised: The upper part is a copy of the input signal.

Original: The lower part consist of the transformed input, where the transformation is a multiplication of (convolutional) weights W i with the input signal, followed by a non-linearity φ having
Revised: The lower part consist of the transformed input, where the transformation is a multiplication of (convolutional) weights W i with the input signal, followed by a non-linearity φ having

Original: Lip( φ ) ≤ 1, such as ReLU, ELU, LipSwish, or tanh.
Revised: Lip( φ ) ≤ 1, such as ReLU, ELU, LipSwish, or tanh.

Original: As an example, a Dense Layer h 2 can be composed as follows:
Revised: As an example, a Dense Layer h 2 can be composed as follows:

Original: [Equation]
Revised: [Equation]

Original: 3.1. Enforcing Lipschitz constraint
Revised: 3.1. Enforcing Lipschitz constraint

Original: If we enforce the function g to satisfy Lip( g ) < 1, the DenseBlock F is invertible and the
Revised: If we enforce the function g to satisfy Lip( g ) < 1, the DenseBlock F is invertible and the

Original: Banach ﬁxed point theorem holds.
Revised: Banach ﬁxed point theorem holds.

Original: As a result, the inverse can be approximated in the same manner as in (Behrmann et al., 2018).
Revised: As a result, the inverse can be approximated in the same manner as in (Behrmann et al., 2018).

Original: To satisfy Lip( g ) < 1, we can enforce Lip( h i ) <for all n layers.
Revised: To satisfy Lip( g ) < 1, we can enforce Lip( h i ) <for all n layers.

Original: Therefore, we ﬁrst need to determine the Lipschitz constant for a Dense
Revised: Therefore, we ﬁrst need to determine the Lipschitz constant for a Dense

Original: Layer h i .
Revised: Layer h i .

Original: We know that a function f is K-Lipschitz if for all points v and w the following holds (for the full derivation see Appendix A):
Revised: We know that a function f is K-Lipschitz if for all points v and w the following holds (for the full derivation see Appendix A):

Original: [Equation]
Revised: [Equation]

Original: where we assume that the distance metrics d X = d Y = d are chosen to be the (cid:96) 2 -norm.
Revised: where we assume that the distance metrics d X = d Y = d are chosen to be the (cid:96) 2 -norm.

Original: Further, let two functions f 1 and f 2 be concatenated in h :
Revised: Further, let two functions f 1 and f 2 be concatenated in h :

Original: [Equation]
Revised: [Equation]

Original: where function f 1 is the upper part and f 2 is the lower part.
Revised: where function f 1 is the upper part and f 2 is the lower part.

Original: We can now ﬁnd an analytical form to express a limit on K for the Dense Layer in the form of Equation (5):
Revised: We can now ﬁnd an analytical form to express a limit on K for the Dense Layer in the form of Equation (5):

Original: [Equation]
Revised: [Equation]

Original: where we know that the Lipschitz constant of h consist of two parts, namely, Lip( f 1 )
Revised: where we know that the Lipschitz constant of h consist of two parts, namely, Lip( f 1 )

Original: =
Revised: =

Original: Kand Lip( f 2 ) =
Revised: Kand Lip( f 2 ) =

Original: K 2 .
Revised: K 2 .

Original: Therefore, the Lipschitz constant of layer h can be expressed as:
Revised: Therefore, the Lipschitz constant of layer h can be expressed as:

Original: [Equation]
Revised: [Equation]

Original: With spectral normalization of Equation (2), we know that we can enforce (convolutional) weights W i to be at most 1-Lipschitz.
Revised: With spectral normalization of Equation (2), we know that we can enforce (convolutional) weights W i to be at most 1-Lipschitz.

Original: Hence, for all n Dense Layers we apply the spectral normalization on the lower part which locally enforces Lip( f 2 ) =
Revised: Hence, for all n Dense Layers we apply the spectral normalization on the lower part which locally enforces Lip( f 2 ) =

Original: K 2 < 1.
Revised: K 2 < 1.

Original: Further, since we enforce each layer
Revised: Further, since we enforce each layer

Original: h i to be at most 1-Lipschitz and we start with h 1 , where f 1 ( x ) =
Revised: h i to be at most 1-Lipschitz and we start with h 1 , where f 1 ( x ) =

Original: x , we know that Lip( f 1 )
Revised: x , we know that Lip( f 1 )

Original: = 1.
Revised: = 1.

Original: Therefore, the Lipschitz constant of an entire layer can be at most Lip( h ) = √ 1 2 + 1 2 = √ 2, thus dividing by this limit enforces each layer to be at most 1-Lipschitz.
Revised: Therefore, the Lipschitz constant of an entire layer can be at most Lip( h ) = √ 1 2 + 1 2 = √ 2, thus dividing by this limit enforces each layer to be at most 1-Lipschitz.

Original: 3.2. Learnable concatenation
Revised: 3.2. Learnable concatenation

Original: We have shown that we can enforce an entire Dense Layer to have Lip( h i ) < 1 by applying a spectral norm on the (convolutional) weights W i and then divide the layer
Revised: We have shown that we can enforce an entire Dense Layer to have Lip( h i ) < 1 by applying a spectral norm on the (convolutional) weights W i and then divide the layer

Original: h
Revised: h

Original: i by √ 2.
Revised: i by √ 2.

Original: To optimize and learn the importance of the concatenated representations, we create learnable parameters η 1 and η 2 for, respectively, the upper and lower part of each layer
Revised: To optimize and learn the importance of the concatenated representations, we create learnable parameters η 1 and η 2 for, respectively, the upper and lower part of each layer

Original: h
Revised: h

Original: i .
Revised: i .

Original: Since the upper and lower part of the layer can be at most 1-Lipschitz, multiplication by these factors results in functions that are at most η 1 -Lipschitz and η 2 -Lipschitz.
Revised: Since the upper and lower part of the layer can be at most 1-Lipschitz, multiplication by these factors results in functions that are at most η 1 -Lipschitz and η 2 -Lipschitz.

Original: From Appendix A we know that the layer is then at most (cid:112) η 21 + η 22 − Lipschitz.
Revised: From Appendix A we know that the layer is then at most (cid:112) η 21 + η 22 − Lipschitz.

Original: Dividing by this factor results in a bound that is at most 1-Lipschitz.
Revised: Dividing by this factor results in a bound that is at most 1-Lipschitz.

Original: In practice, we initialize η 1 and η 2 at value 1 and during training use a softplus function to avoid them being negative.
Revised: In practice, we initialize η 1 and η 2 at value 1 and during training use a softplus function to avoid them being negative.

Original: The range of the normalized parameters is between 0 ≤ ˆ η 1 , ˆ η 2 ≤ 1 and can be expressed on the unit circle as is shown in Figure 1.
Revised: The range of the normalized parameters is between 0 ≤ ˆ η 1 , ˆ η 2 ≤ 1 and can be expressed on the unit circle as is shown in Figure 1.

Original: In the special case where η 1 = η 2 , the normalized parameters are ˆ η 1 = ˆ η 2 = 12 √ 2.
Revised: In the special case where η 1 = η 2 , the normalized parameters are ˆ η 1 = ˆ η 2 = 12 √ 2.

Original: This case corresponds to the situation in Section 3.1 where the concatenation was not learned.
Revised: This case corresponds to the situation in Section 3.1 where the concatenation was not learned.

Original: An additional advantage is that the normalized ˆ η 1 and ˆ η 2 express the importance of the upper and lower signal.
Revised: An additional advantage is that the normalized ˆ η 1 and ˆ η 2 express the importance of the upper and lower signal.

Original: For example, when ˆ η 1 > ˆ η 2 , the input signal is of more importance than the transformed signal.
Revised: For example, when ˆ η 1 > ˆ η 2 , the input signal is of more importance than the transformed signal.

Original: 4. Experiments
Revised: 4. Experiments

Original: To make a clear comparison between the performance of Residual Flows and i-DenseNets, we train both models on 2-dimensional toy data and on high-dimensional image data, MNIST and CIFAR10.
Revised: To make a clear comparison between the performance of Residual Flows and i-DenseNets, we train both models on 2-dimensional toy data and on high-dimensional image data, MNIST and CIFAR10.

Original: To benchmark the models, we use the architecture of Residual ﬂow (Chen et al., 2019).
Revised: To benchmark the models, we use the architecture of Residual ﬂow (Chen et al., 2019).

Original: Since we have a constrained computational budget, we use a smaller architecture of the model and choose number of scales for the toy data and image data set to, respectively, 10 blocks and 4 blocks per 3 scales instead of 100 blocks and 16 blocks perscales.
Revised: Since we have a constrained computational budget, we use a smaller architecture of the model and choose number of scales for the toy data and image data set to, respectively, 10 blocks and 4 blocks per 3 scales instead of 100 blocks and 16 blocks perscales.

Original: For the other arguments, default settings are used.
Revised: For the other arguments, default settings are used.

Original: To compare Residual Flows with i-DenseNets, we utilize an architecture that uses a similar number of parameters for each dataset trained on.
Revised: To compare Residual Flows with i-DenseNets, we utilize an architecture that uses a similar number of parameters for each dataset trained on.

Original: A detailed description of this architecture can be found in Appendix B.
Revised: A detailed description of this architecture can be found in Appendix B.

Original: Furthermore, we add the option to learn the parameters of the concatenation.
Revised: Furthermore, we add the option to learn the parameters of the concatenation.

Original: The models trained on toy data were trained for 50,000 iterations (default setting) and on image data for 200 epochs.
Revised: The models trained on toy data were trained for 50,000 iterations (default setting) and on image data for 200 epochs.

Original: 4.1. Toy data
Revised: 4.1. Toy data

Original: We trained the models on diﬀerent types of 2-dimensional toy data distributions, namely, two circles, a checkerboard, and two moons.
Revised: We trained the models on diﬀerent types of 2-dimensional toy data distributions, namely, two circles, a checkerboard, and two moons.

Original: The results of the learned density distributions are presented in Figure 2( a ).
Revised: The results of the learned density distributions are presented in Figure 2( a ).

Original: We observe that Residual Flows are capable to capture highprobability areas.
Revised: We observe that Residual Flows are capable to capture highprobability areas.

Original: However, they have trouble with learning low probability regions for two circles and moons.
Revised: However, they have trouble with learning low probability regions for two circles and moons.

Original: i-DenseNets are capable in capturing all regions of the datasets.
Revised: i-DenseNets are capable in capturing all regions of the datasets.

Original: Table 1, where the negative log-likelihood for the models are presented, also shows that i-DenseNets with and without learnable concatenation (LC) outperform Residual Flows.
Revised: Table 1, where the negative log-likelihood for the models are presented, also shows that i-DenseNets with and without learnable concatenation (LC) outperform Residual Flows.

Original: The biggest diﬀerence in performance is for two moons where i-DenseNets with LC obtain 2 .
Revised: The biggest diﬀerence in performance is for two moons where i-DenseNets with LC obtain 2 .

Original: 39 nats compared to 2 .
Revised: 39 nats compared to 2 .

Original: 60 nats for Residual Flows.
Revised: 60 nats for Residual Flows.

Original: This is consistent with the density estimation plots where i-DenseNets are better in capturing the data distribution than Residual Flows.
Revised: This is consistent with the density estimation plots where i-DenseNets are better in capturing the data distribution than Residual Flows.

Original: [Figure]
Revised: [Figure]

Original:  results after 50, iterations of the Residual Flow and iDenseNet.
Revised: ) Density estimation results after 50, iterations of the Residual Flow and iDenseNet.

Original: Trained on 2-dimensional 
Revised: Trained on 2-dimensional toy data.

Original: [Figure]
Revised: [Figure]

Original: data
Revised: learnable concatenation

Original: [Figure]
Revised: [Figure]

Original: 4.2. Image Data
Revised: 4.2. Image Data

Original: The results of the models trained on MNIST and CIFAR10 data are presented in Table 1.
Revised: The results of the models trained on MNIST and CIFAR10 data are presented in Table 1.

Original: We notice that i-DenseNets outperform Residual Flows in bits per dimension (bpd) on CIFAR10 with 3 .
Revised: We notice that i-DenseNets outperform Residual Flows in bits per dimension (bpd) on CIFAR10 with 3 .

Original: 41 bpd without LC and 3 . 

Revised: 41 bpd without LC and 3 .

Original: bpd with LC, against 3 .
Revised: bpd with LC, against 3 .

Original: 42 bpd for the Residual
Revised: 42 bpd for the Residual

Original: Flow.
Revised: Flow.

Original: Figure 2( b ) presents samples of the i-DenseNet with LC trained on CIFAR10, in Ap- pendix D more samples of the models can be found.
Revised: Figure 2( b ) presents samples of the i-DenseNet with LC trained on CIFAR10, in Ap- pendix D more samples of the models can be found.

Original: Additionally, in Appendix C a heatmap of the normalized parameters ˆ η 1 and ˆ η 2 of the learnable concatenation is presented.
Revised: Additionally, in Appendix C a heatmap of the normalized parameters ˆ η 1 and ˆ η 2 of the learnable concatenation is presented.

Original: During training on MNIST the original Residual Flow suﬀered from unstable results.
Revised: During training on MNIST the original Residual Flow suﬀered from unstable results.

Original: This might be due to the coeﬃcient for the spectral normalization, which controls the
Revised: This might be due to the coeﬃcient for the spectral normalization, which controls the

Original: Lipschitz constraint.
Revised: Lipschitz constraint.

Original: In return, this leads to an unstable Jacobian determinant estimation.
Revised: In return, this leads to an unstable Jacobian determinant estimation.

Original: We adjusted the Lipschitz coeﬃcient for the spectral normalization by setting it to 0 .for
Revised: We adjusted the Lipschitz coeﬃcient for the spectral normalization by setting it to 0 .for

Original: all models.
Revised: all models.

Original: Additionally, the concatenation in DenseNets is multiplied by 0 .
Revised: Additionally, the concatenation in DenseNets is multiplied by 0 .

Original: Due to slight ﬂuctuations, the results are averaged over the last 5 epochs, which are presented in Table 1.
Revised: Due to slight ﬂuctuations, the results are averaged over the last 5 epochs, which are presented in Table 1.

Original: We observe that i-DeseNets without and with LC outperform the Residual Flow with respectively 1 . 

Revised: We observe that i-DeseNets without and with LC outperform the Residual Flow with respectively 1 .

Original: bpd and 1 . 04 bpd against 1 .
Revised: bpd and 1 . 04 bpd against 1 .

Original: 08 bpd of the Residual Flow.
Revised: 08 bpd of the Residual Flow.

Original: In general we observe that i-DenseNets with LC outperform Residual Flows and i-DenseNets without
Revised: In general we observe that i-DenseNets with LC outperform Residual Flows and i-DenseNets without

Original: LC.
Revised: LC.

Original: On two moons, the performance of i-DenseNets with and without LC are tied.
Revised: On two moons, the performance of i-DenseNets with and without LC are tied.

Original: [Table]
Revised: [Table]

Original: 5. Conclusion
Revised: 5. Conclusion

Original: We introduced i-DenseNets, a parameter eﬃcient alternative to Invertible ResNets.
Revised: We introduced i-DenseNets, a parameter eﬃcient alternative to Invertible ResNets.

Original: Our method enforces invertibility by satisfying the Lipschitz continuity in Dense Layers.
Revised: Our method enforces invertibility by satisfying the Lipschitz continuity in Dense Layers.

Original: In addition, we proposed a version where the concatenation is learned during training, which also indicates which representations are used.
Revised: In addition, we proposed a version where the concatenation is learned during training, which also indicates which representations are used.

Original: We used a smaller architecture under an equal parameter budget, where we demonstrated the performance of i-DenseNets and compared these models to Residual Flows on toy, MNIST, and CIFAR10 data.
Revised: We used a smaller architecture under an equal parameter budget, where we demonstrated the performance of i-DenseNets and compared these models to Residual Flows on toy, MNIST, and CIFAR10 data.

Original: In conclusion, both iDenseNets with ﬁxed and learnable concatenation outperformed Residual Flows in negative log-likelihood.
Revised: In conclusion, both iDenseNets with ﬁxed and learnable concatenation outperformed Residual Flows in negative log-likelihood.

Original: Appendix A. Derivation of Lipschitz constant K for the concatenation
Revised: Appendix A. Derivation of Lipschitz constant K for the concatenation

Original: We know that a function f is K-Lipschitz if for all points v and w the following holds:
Revised: We know that a function f is K-Lipschitz if for all points v and w the following holds:

Original: [Equation]
Revised: [Equation]

Original: where d Y and d X are distance metrics and K is the Lipschitz constant.
Revised: where d Y and d X are distance metrics and K is the Lipschitz constant.

Original: Consider the case where we assume to have the same distance metric d Y =
Revised: Consider the case where we assume to have the same distance metric d Y =

Original: d X = d and where the distance metric is assumed to be chosen as any p -norm, where p ≥ 1, for vectors:|| δ [Equation] =
Revised: d X = d and where the distance metric is assumed to be chosen as any p -norm, where p ≥ 1, for vectors:

Original: [Equation]
Revised: [Equation]

Original: | δ i | p .
Revised: | δ i | p .

Original: Further, we assume a DenseBlock to be a function h where the output for each data point v and w is expressed as follows:
Revised: Further, we assume a DenseBlock to be a function h where the output for each data point v and w is expressed as follows:

Original: [Equation]
Revised: [Equation]

Original: where in this paper for a Dense Layer and for a data point x the function f 1 ( x ) = x and fexpresses a linear combination of (convolutional) weights with x followed by a non-linearity, for example φ ( W 1 x ).
Revised: where in this paper for a Dense Layer and for a data point x the function f 1 ( x ) = x and fexpresses a linear combination of (convolutional) weights with x followed by a non-linearity, for example φ ( W 1 x ).

Original: We can re-write Equation (9) for the DenseNet function as:
Revised: We can re-write Equation (9) for the DenseNet function as:

Original: [Equation]
Revised: [Equation]

Original: where K is the unknown Lipschitz constant for the entire DenseBlock.
Revised: where K is the unknown Lipschitz constant for the entire DenseBlock.

Original: However, we can ﬁnd an analytical form to express a limit on K. To solve this, we know that the distance between h v and h w can be expressed by the p -norm as:
Revised: However, we can ﬁnd an analytical form to express a limit on K. To solve this, we know that the distance between h v and h w can be expressed by the p -norm as:

Original: [Equation]
Revised: [Equation]

Original: where we can simplify the equation by taking the p -th power:
Revised: where we can simplify the equation by taking the p -th power:

Original: [Equation]
Revised: [Equation]

Original: Since we know that the distance of a can be expressed as:
Revised: Since we know that the distance of a can be expressed as:

Original: [Equation]
Revised: [Equation]

Original: which is similar for the distance of b , re-writing the second term of Equation (13) in the form of Equation (11)
Revised: which is similar for the distance of b , re-writing the second term of Equation (13) in the form of Equation (11)

Original: is assumed to be of form:
Revised: is assumed to be of form:

Original: [Equation]
Revised: [Equation]

Original: which is similar for b , d ( b v , b w ) p ≤ K
Revised: which is similar for b , d ( b v , b w ) p ≤ K

Original: p 2 d ( v, w ) p .
Revised: p 2 d ( v, w ) p .

Original: Assuming this, we can ﬁnd a form of
Revised: Assuming this, we can ﬁnd a form of

Original: Equation (11) by substituting with Equation (13) and Equation (15):
Revised: Equation (11) by substituting with Equation (13) and Equation (15):

Original: [Equation]
Revised: [Equation]

Original: Now, taking the p -th root we have:
Revised: Now, taking the p -th root we have:

Original: [Equation]
Revised: [Equation]

Original: where we have derived the form of Equation (11) and where Lip( h )
Revised: where we have derived the form of Equation (11) and where Lip( h )

Original: =
Revised: =

Original: K is expressed as:
Revised: K is expressed as:

Original: [Equation]
Revised: [Equation]

Original: where Lip( f 1 ) =
Revised: where Lip( f 1 ) =

Original: K 1 and Lip( f 2 )
Revised: K 1 and Lip( f 2 )

Original: =
Revised: =

Original: K 2 , which are assumed to be known Lipschitz constants.
Revised: K 2 , which are assumed to be known Lipschitz constants.

Original: Appendix B. Implementation
Revised: Appendix B. Implementation

Original: We used a smaller architecture of Residual Flows (Chen et al., 2019), with an adjustment of number of blocks per scale set to 4 instead of 16.
Revised: We used a smaller architecture of Residual Flows (Chen et al., 2019), with an adjustment of number of blocks per scale set to 4 instead of 16.

Original: For training we ensured an equal parameter budget for i-DenseNets.
Revised: For training we ensured an equal parameter budget for i-DenseNets.

Original: The architecture of i-DenseNets for image data are presented in Table 2.
Revised: The architecture of i-DenseNets for image data are presented in Table 2.

Original: A DenseBlock consist of several Dense Layers.
Revised: A DenseBlock consist of several Dense Layers.

Original: The last Dense Layer h n is followed by a 1 × 1 convolution to match the output of size R d after which a squeezing layer is applied.
Revised: The last Dense Layer h n is followed by a 1 × 1 convolution to match the output of size R d after which a squeezing layer is applied.

Original: The ﬁnal part of the network consist of a Fully Connected (FC) layer with number of blocks set to 4.
Revised: The ﬁnal part of the network consist of a Fully Connected (FC) layer with number of blocks set to 4.

Original: Before the concatenation in the FC layer, a Linear layer of input
Revised: Before the concatenation in the FC layer, a Linear layer of input

Original: R d to output dimension 64 is applied, followed by the Dense Layer with DenseNet growth 32 and activation LipSwish.
Revised: R d to output dimension 64 is applied, followed by the Dense Layer with DenseNet growth 32 and activation LipSwish.

Original: The DenseNet depth is set to 3.
Revised: The DenseNet depth is set to 3.

Original: The ﬁnal part consist of a
Revised: The ﬁnal part consist of a

Original: Linear layer to match the output of size R d .
Revised: Linear layer to match the output of size R d .

Original: Nr.
Revised: Nr.

Original: of scales
Revised: of scales

Original: Nr.
Revised: Nr.

Original: of blocks per scale
Revised: of blocks per scale

Original: DenseNet
Revised: DenseNet

Original: Depth
Revised: Depth

Original: DenseNet
Revised: DenseNet

Original: Growth
Revised: Growth

Original: Dense Layer
Revised: Dense Layer

Original: Output   108 (MNIST) 124 (CIFAR10)   3 ×conv
Revised: Output   108 (MNIST) 124 (CIFAR10)   3 ×conv

Original: LipSwish concat   (cid:2) 1 ×conv (cid:3)
Revised: LipSwish concat   (cid:2) 1 ×conv (cid:3)

Original: Toy data
Revised: Toy data

Original: We used 10 scale blocks for all models.
Revised: We used 10 scale blocks for all models.

Original: Furthermore, we used default settings of
Revised: Furthermore, we used default settings of

Original: Residual Flows.
Revised: Residual Flows.

Original: For i-DenseNets, we choose a DenseNet -depth and -growth of, respectively, 4 and 90 with 504K parameters and Residual Flows utilize 501K parameters.
Revised: For i-DenseNets, we choose a DenseNet -depth and -growth of, respectively, 4 and 90 with 504K parameters and Residual Flows utilize 501K parameters.

Original: MNIST
Revised: MNIST

Original: All models used 3 scales where the number of blocks per scale is set to 4.
Revised: All models used 3 scales where the number of blocks per scale is set to 4.

Original: Due to instability of Residual Flows, we set our coeﬃcient that controls the Lipschitz constraint from 0 .
Revised: Due to instability of Residual Flows, we set our coeﬃcient that controls the Lipschitz constraint from 0 .

Original: 98 to 0 .
Revised: 98 to 0 .

Original: Furthermore, default settings of Residual Flows are used.
Revised: Furthermore, default settings of Residual Flows are used.

Original: For i-DenseNets, we used a coeﬃcient controlling the Lipschitz of the concatenated blocks set to 0 .
Revised: For i-DenseNets, we used a coeﬃcient controlling the Lipschitz of the concatenated blocks set to 0 .

Original: i-DenseNets use a DenseNet -depth and -growth of, respectively, 3 and 108 with 8 . {+
+}
Revised: i-DenseNets use a DenseNet -depth and -growth of, respectively, 3 and 108 with 5 . {+
+}

Original: 7M parameters and Residual Flows utilize 8 . {+
+}
Revised: M parameters and Residual Flows utilize 5 . {+
+}

Original: 7M parameters.
Revised: M parameters.

Original: CIFAR
Revised: CIFAR

Original: All models used 3 scales where the number of blocks per scale is set to 4.
Revised: All models used 3 scales where the number of blocks per scale is set to 4.

Original: Furthermore, default settings of Residual Flows are used.
Revised: Furthermore, default settings of Residual Flows are used.

Original: i-DenseNets use a DenseNetdepth and -growth of, respectively, 3 and 124 with 8 .
Revised: i-DenseNets use a DenseNetdepth and -growth of, respectively, 3 and 124 with 8 .

Original: 7M parameters and Residual Flows utilize the 8 .
Revised: 7M parameters and Residual Flows utilize the 8 .

Original: 7M parameters.
Revised: 7M parameters.

Original: Appendix C. Visualization of learnable concatenation
Revised: Appendix C. Visualization of learnable concatenation

Original: h 1 a h 1 b
Revised: h 1 a h 1 b

Original: h 1 c
Revised: h 1 c

Original: h 2 a h 2 b h 2 c
Revised: h 2 a h 2 b h 2 c

Original: h 3 a h 3 b h 3 c h 4 a h4 b h 4 c
Revised: h 3 a h 3 b h 3 c h [Table] a h

Revised: [Table]

Revised: h 4 c

Original: Scale
Revised: Scale

Original: Scale Scale
Revised: Scale Scale

Original: FC 0.
Revised: FC 0.

Original: [Figure]
Revised: [Figure]

Original: Heatmap of ˆ ηh1 a h 1 b h 1 c
Revised: Heatmap of ˆ ηh

Original: h 2 a h 2 b h 2 c
Revised: [Table]

Original: h 3 a h 3 b h 3 c h 4 a h 4 b h 4 c 
Revised: a h 1 b h 1 c h 2 a h 2 b h 2 c h 3 a h

Original: Scale
Revised: [Table]

Original: Scale
Revised: b h 3 c h [Table] a h [Table]

Original: Scale FC 0.
Revised: h

Original: [Figure]
Revised: [Table]

Original: [Figure] [Figure] [Figure] [Figure]
Revised: c

Original: [Figure]
Revised: Scale

Original: [Figure] [Figure]
Revised: Scale Scale

Original: [Figure] [Figure] [Figure] [Figure]
Revised: FC 0.

Original: [Figure]
Revised: [Figure]

Original: Figure 3 shows the heatmap for ( a ) the normalized parameter ˆ η 1 and ( b ) normalized parameter ˆ η 2 after 200 epochs, trained on CIFAR10.
Revised: Figure 3 shows the heatmap for ( a ) the normalized parameter ˆ η 1 and ( b ) normalized parameter ˆ η 2 after 200 epochs, trained on CIFAR10.

Original: Every scale level 1, 2 and 3 contain
Revised: Every scale level 1, 2 and 3 contain

Original: DenseBlocks, that each contain 3 Dense Layers with convolutional layers.
Revised: DenseBlocks, that each contain 3 Dense Layers with convolutional layers.

Original: The ﬁnal level
Revised: The ﬁnal level

Original: FC indicates that fully connected layers are used.
Revised: FC indicates that fully connected layers are used.

Original: The letters ‘a’, ‘b’, and ‘c’ index the
Revised: The letters ‘a’, ‘b’, and ‘c’ index the

Original: Dense Layers per block.
Revised: Dense Layers per block.

Original: Remarkably, all scale levels for the last layers h ic give little importance to the input signal.
Revised: Remarkably, all scale levels for the last layers h ic give little importance to the input signal.

Original: The input signals for these layers are in most cases multiplied with ˆ η 1 (close to) zero, while the transformed signal uses almost all the information when multiplied with ˆ η 2 which is close to one.
Revised: The input signals for these layers are in most cases multiplied with ˆ η 1 (close to) zero, while the transformed signal uses almost all the information when multiplied with ˆ η 2 which is close to one.

Original: This indicates that the transformed signal is of more importance for the network than the input signal.
Revised: This indicates that the transformed signal is of more importance for the network than the input signal.

Original: For the fully connected part, this diﬀerence is not that pronounced.
Revised: For the fully connected part, this diﬀerence is not that pronounced.

Original: Appendix D. Model samples
Revised: Appendix D. Model samples

Original: This appendix contains real images and samples of the models trained on MNIST (Figure 4).
Revised: This appendix contains real images and samples of the models trained on MNIST (Figure 4).

Original: ( a ) shows real images of MNIST, ( b ) shows samples of the Residual Flow trained on MNIST, as well as samples of ( c ) i-DenseNet without LC and ( d ) i-DenseNet with LC.
Revised: ( a ) shows real images of MNIST, ( b ) shows samples of the Residual Flow trained on MNIST, as well as samples of ( c ) i-DenseNet without LC and ( d ) i-DenseNet with LC.

Original: Figure 5 contains real images and samples of the models trained on images of CIFAR10.
Revised: Figure 5 contains real images and samples of the models trained on images of CIFAR10.

Original: ( a ) shows real images of CIFAR10, ( b ) shows samples of the Residual Flow trained on
Revised: ( a ) shows real images of CIFAR10, ( b ) shows samples of the Residual Flow trained on

Original: CIFAR10, as well as samples of ( c ) i-DenseNet without LC and ( d ) i-DenseNet with LC.
Revised: CIFAR10, as well as samples of ( c ) i-DenseNet without LC and ( d ) i-DenseNet with LC.

Original: [Figure]
Revised: [Figure]

Original:  c ) Samples of i-DenseNet without LC.
Revised: ( c ) Samples of i-DenseNet without LC.

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original:  of i-DenseNet without LC
Revised: Samples of i-DenseNet without LC.

Original: [Figure]
Revised: [Figure]