Original: C ONDITIONAL I NVERTIBLE N EURAL N ETWORKS
Revised: C ONDITIONAL I NVERTIBLE N EURAL N ETWORKS

Original: F OR G UIDED I MAGE G ENERATION
Revised: F OR G UIDED I MAGE G ENERATION

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: In this work, we address the task of natural image generation guided by a conditioning input.
Revised: In this work, we address the task of natural image generation guided by a conditioning input.

Original: We introduce a new architecture called conditional invertible neural network (cINN).
Revised: We introduce a new architecture called conditional invertible neural network (cINN).

Original: It combines the purely generative INN model with an unconstrained feed-forward network, which efﬁciently preprocesses the conditioning input into useful features.
Revised: It combines the purely generative INN model with an unconstrained feed-forward network, which efﬁciently preprocesses the conditioning input into useful features.

Original: All parameters of a cINN are jointly optimized with a stable, maximum likelihood-based training procedure.
Revised: All parameters of a cINN are jointly optimized with a stable, maximum likelihood-based training procedure.

Original: Even though INNs and other normalizing ﬂow models have received very little attention in the literature in contrast to GANs, we ﬁnd that cINNs can achieve comparable quality, with some remarkable properties absent in cGANs, e.g. apparent immunity to mode collapse.
Revised: Even though INNs and other normalizing ﬂow models have received very little attention in the literature in contrast to GANs, we ﬁnd that cINNs can achieve comparable quality, with some remarkable properties absent in cGANs, e.g. apparent immunity to mode collapse.

Original: We demonstrate these properties for the tasks of MNIST digit generation and image colorization.
Revised: We demonstrate these properties for the tasks of MNIST digit generation and image colorization.

Original: Furthermore, we take advantage of our bidirectional cINN architecture to explore and manipulate emergent properties of the latent space, such as changing the image style in an intuitive way.
Revised: Furthermore, we take advantage of our bidirectional cINN architecture to explore and manipulate emergent properties of the latent space, such as changing the image style in an intuitive way.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Generative adversarial networks (GANs) produce ever larger and more realistic samples (Karras et al., 2017; Brock et al., 2019).
Revised: Generative adversarial networks (GANs) produce ever larger and more realistic samples (Karras et al., 2017; Brock et al., 2019).

Original: Hence they have become the primary choice for a majority of image generation tasks.
Revised: Hence they have become the primary choice for a majority of image generation tasks.

Original: As such, their conditional variants (cGANs) would appear to be the natural tool for conditional image generation as well, and they have successfully been applied in many scenarios (Ledig et al., 2017; Miyato & Koyama, 2018).
Revised: As such, their conditional variants (cGANs) would appear to be the natural tool for conditional image generation as well, and they have successfully been applied in many scenarios (Ledig et al., 2017; Miyato & Koyama, 2018).

Original: However, a lack in diversity is especially common when the condition itself is an image, and special precautions have to be taken to avoid mode collapse and training stability continues to pose a challenge.
Revised: However, a lack in diversity is especially common when the condition itself is an image, and special precautions have to be taken to avoid mode collapse and training stability continues to pose a challenge.

Original: Conditional variational autoencoders (cVAEs) do not suffer from the same problems.
Revised: Conditional variational autoencoders (cVAEs) do not suffer from the same problems.

Original: Training is generally stable, and since every data point is assigned a region in latent space, sampling yields the full variety of data seen during training.
Revised: Training is generally stable, and since every data point is assigned a region in latent space, sampling yields the full variety of data seen during training.

Original: However cVAEs come with drawbacks of their own: The assumption of a Gaussian posterior on the decoder side implies an L2 reconstruction loss, which is known to cause blurriness.
Revised: However cVAEs come with drawbacks of their own: The assumption of a Gaussian posterior on the decoder side implies an L2 reconstruction loss, which is known to cause blurriness.

Original: In addition, the partition of the latent space into diagonal Gaussians leads to either mode-mixing issues or regions of poor sample quality (Kingma et al., 2016).
Revised: In addition, the partition of the latent space into diagonal Gaussians leads to either mode-mixing issues or regions of poor sample quality (Kingma et al., 2016).

Original: There has also been some success in combining aspects of both approaches for certain tasks, such as (Isola et al., 2017; Zhu et ., 2017b
Revised: There has also been some success in combining aspects of both approaches for certain tasks, such as (Isola et al., 2017; Zhu et al., 2017b; Park et al., 2019).

Original: We propose a third approach, by extending Invertible Neural Networks (INNs, Dinh et al.
Revised: We propose a third approach, by extending Invertible Neural Networks (INNs, Dinh et al.

Original: ; Kingma & Dhariwal (2018); Ardizzone et al.
Revised: ; Kingma & Dhariwal (2018); Ardizzone et al.

Original: for the task of conditional image generation, by adding conditioning inputs to their core building blocks.
Revised: for the task of conditional image generation, by adding conditioning inputs to their core building blocks.

Original: INNs are neural networks which are by construction bijective, efﬁciently invertible, and have a tractable Jacobian determinant.
Revised: INNs are neural networks which are by construction bijective, efﬁciently invertible, and have a tractable Jacobian determinant.

Original: They represent transport maps between the input distribution p ( x ) and a prescribed, easy-to-samplefrom latent distribution p ( z ) .
Revised: They represent transport maps between the input distribution p ( x ) and a prescribed, easy-to-samplefrom latent distribution p ( z ) .

Original: During training, the likelihood of training samples from p ( x ) is maximized in latent space, while at inference time, z -samples can trivially be transformed back to the data domain.
Revised: During training, the likelihood of training samples from p ( x ) is maximized in latent space, while at inference time, z -samples can trivially be transformed back to the data domain.

Original: Previously, INNs have been used successfully for unconditional image generation, e.g. by Dinh et al.
Revised: Previously, INNs have been used successfully for unconditional image generation, e.g. by Dinh et al.

Original: and Kingma & Dhariwal (2018).
Revised: and Kingma & Dhariwal (2018).

Original: Unconditional INN training is related to that of VAEs, but it compensates for some key disadvantages:
Revised: Unconditional INN training is related to that of VAEs, but it compensates for some key disadvantages:

Original: Firstly, since reconstructions are perfect by design, no reconstruction loss is needed, and generated images do not become blurry.
Revised: Firstly, since reconstructions are perfect by design, no reconstruction loss is needed, and generated images do not become blurry.

Original: Secondly, each x maps to exactly one z in latent space, and there is no need for posteriors p ( z | x ) .
Revised: Secondly, each x maps to exactly one z in latent space, and there is no need for posteriors p ( z | x ) .

Original: This avoids the VAE problem of disjoint or overlapping regions in latent space.
Revised: This avoids the VAE problem of disjoint or overlapping regions in latent space.

Original: In terms of training stability and sample diversity, INNs show the same strengths as autoencoder architectures, but with superior image quality.
Revised: In terms of training stability and sample diversity, INNs show the same strengths as autoencoder architectures, but with superior image quality.

Original: We ﬁnd that these positive aspects apply to conditional INNs (cINNs) as well.
Revised: We ﬁnd that these positive aspects apply to conditional INNs (cINNs) as well.

Original: One limitation of INNs is that their design restricts the use of some standard components of neural networks, such as pooling and batch normalization layers.
Revised: One limitation of INNs is that their design restricts the use of some standard components of neural networks, such as pooling and batch normalization layers.

Original: Our conditional architecture alleviates this problem, as the conditional inputs can be preprocessed by a conditioning network with a standard feed-forward architecture, which can be learned jointly with the cINN to greatly improve its generative capabilities.
Revised: Our conditional architecture alleviates this problem, as the conditional inputs can be preprocessed by a conditioning network with a standard feed-forward architecture, which can be learned jointly with the cINN to greatly improve its generative capabilities.

Original: We demonstrate the quality of cINNs for conditional image generation and uncover emergent properties of the latent space, for the tasks of conditional MNIST generation and diverse colorization of ImageNet.
Revised: We demonstrate the quality of cINNs for conditional image generation and uncover emergent properties of the latent space, for the tasks of conditional MNIST generation and diverse colorization of ImageNet.

Original: Given this, we believe that the cINN architecture brings the research ﬁeld of INNs and other normalizing ﬂow models a substantial step forward.
Revised: Given this, we believe that the cINN architecture brings the research ﬁeld of INNs and other normalizing ﬂow models a substantial step forward.

Original: Despite the fact that INNs have received very little attention in the literature in contrast to GANs, we ﬁnd that cINNs can achieve comparable quality to cGANs, with some remarkable properties absent in cGANs.
Revised: Despite the fact that INNs have received very little attention in the literature in contrast to GANs, we ﬁnd that cINNs can achieve comparable quality to cGANs, with some remarkable properties absent in cGANs.

Original: This includes diverse outputs by default, compared to sparse support or mode collapse in cGANs, easier explainability and interpretability of the learned latent representation, as well as simple and intuitive manipulation and interpolation of generated or existing images.
Revised: This includes diverse outputs by default, compared to sparse support or mode collapse in cGANs, easier explainability and interpretability of the learned latent representation, as well as simple and intuitive manipulation and interpolation of generated or existing images.

Original: Our work makes the following contributions:
Revised: Our work makes the following contributions:

Original: • We propose a new architecture called conditional invertible neural network (cINN), which combines an INN with an unconstrained feed-forward network for conditioning.
Revised: • We propose a new architecture called conditional invertible neural network (cINN), which combines an INN with an unconstrained feed-forward network for conditioning.

Original: It generates diverse images with high realism on par with existing approaces, while adding some noteworthy and useful properties.
Revised: It generates diverse images with high realism on par with existing approaces, while adding some noteworthy and useful properties.

Original: • We demonstrate a stable, maximum likelihood-based training procedure for jointly optimizing the parameters of the INN and the conditioning network.
Revised: • We demonstrate a stable, maximum likelihood-based training procedure for jointly optimizing the parameters of the INN and the conditioning network.

Original: • We take advantage of our bidirectional cINN architecture to explore and manipulate emergent properties of the latent space.
Revised: • We take advantage of our bidirectional cINN architecture to explore and manipulate emergent properties of the latent space.

Original: We illustrate this for MNIST digit generation and image colorization.
Revised: We illustrate this for MNIST digit generation and image colorization.

Original:  R ELATED WORK
Revised:  R ELATED WORK

Original: Conditional Generative Modeling.
Revised: Conditional Generative Modeling.

Original: Modern generative models learn to transform noise (usually sampled from multivariate Gaussians) into desired target distributions.
Revised: Modern generative models learn to transform noise (usually sampled from multivariate Gaussians) into desired target distributions.

Original: Methods differ by the modelfamily these transformations are picked from and by the losses determining optimal solutions.
Revised: Methods differ by the modelfamily these transformations are picked from and by the losses determining optimal solutions.

Original: Conditional generative adversarial networks (cGANs, Mirza & Osindero (2014)) train a pair of neural networks: a generator transforms a pair of conditioning and noise vectors to images, and a discriminator penalizes unrealistic looking images.
Revised: Conditional generative adversarial networks (cGANs, Mirza & Osindero (2014)) train a pair of neural networks: a generator transforms a pair of conditioning and noise vectors to images, and a discriminator penalizes unrealistic looking images.

Original: The conditioning information is either concatenated to the noise (Mirza & Osindero, 2014), or fed into the network via conditional batch-norm layers (Dumoulin et al., 2017; Huang & Belongie, 2017; Park et al., 2019).
Revised: The conditioning information is either concatenated to the noise (Mirza & Osindero, 2014), or fed into the network via conditional batch-norm layers (Dumoulin et al., 2017; Huang & Belongie, 2017; Park et al., 2019).

Original: Ensuring diversity of the generated images (for ﬁxed conditioning) appears to be challenging in this approach.
Revised: Ensuring diversity of the generated images (for ﬁxed conditioning) appears to be challenging in this approach.

Original: Recent
Revised: Recent

Original: BigGANs (Brock et al., 2019) successfully address this problem by using very large networks and batch sizes, but require parallel training on up to 512 TPUs.
Revised: BigGANs (Brock et al., 2019) successfully address this problem by using very large networks and batch sizes, but require parallel training on up to 512 TPUs.

Original: PacGANs (Lin et al., 2018) employ augmented discriminators, which evaluate entire batches of real or generated images together rather than one image at a time.
Revised: PacGANs (Lin et al., 2018) employ augmented discriminators, which evaluate entire batches of real or generated images together rather than one image at a time.

Original: CausalGANs
Revised: CausalGANs

Original: (Kocaoglu et al., 2017) train two additional discriminator networks, called “labeler” and “anti-labeler”, with the latter explicitly penalizing the lack of diversity.
Revised: (Kocaoglu et al., 2017) train two additional discriminator networks, called “labeler” and “anti-labeler”, with the latter explicitly penalizing the lack of diversity.

Original: Pix2pix (Isola et al., 2017) addresses the important special case when the target is conditioned on an image in a different modality, e.g. to generate satellite images from maps.
Revised: Pix2pix (Isola et al., 2017) addresses the important special case when the target is conditioned on an image in a different modality, e.g. to generate satellite images from maps.

Original: In addition to the discriminator loss, it minimizes the L1 distance between generated and ground-truth targets using a paired training set, which contains corresponding images from both modalities.
Revised: In addition to the discriminator loss, it minimizes the L1 distance between generated and ground-truth targets using a paired training set, which contains corresponding images from both modalities.

Original: This leads to impressive image quality, but lack of diversity seems to be an especially hard problem in this case.
Revised: This leads to impressive image quality, but lack of diversity seems to be an especially hard problem in this case.

Original: In contrast, our method does not need explicit precautions to promote diversity.
Revised: In contrast, our method does not need explicit precautions to promote diversity.

Original: Bidirectional architectures augment generator networks with complementary encoder networks that learn the generator’s inverse and enable reconstruction losses, which exploit cycle consistency requirements.
Revised: Bidirectional architectures augment generator networks with complementary encoder networks that learn the generator’s inverse and enable reconstruction losses, which exploit cycle consistency requirements.

Original: Conditional variational autoencoders (cVAEs, Sohn et al. (2015)) replace all distributions in a standard VAE (Kingma & Welling, 2013) by the appropriate conditional distributions, and are trained to minimize the evidence lower bound (ELBO loss).
Revised: Conditional variational autoencoders (cVAEs, Sohn et al. (2015)) replace all distributions in a standard VAE (Kingma & Welling, 2013) by the appropriate conditional distributions, and are trained to minimize the evidence lower bound (ELBO loss).

Original: Since variational distributions are typically
Revised: Since variational distributions are typically

Original: Gaussian, the reconstruction penalty is equivalent to squared loss, resulting in rather blurry generated images.
Revised: Gaussian, the reconstruction penalty is equivalent to squared loss, resulting in rather blurry generated images.

Original: This is avoided by AGE networks (Ulyanov et al., 2018) and CycleGANs (Zhu et al., 2017a), which combine standard cGAN discriminators with L1 reconstruction loss in the data domain, and bidirectional conditional GANs (Jaiswal et al., 2017), which extend the GAN discriminator to act on the distributions in data and latent space jointly.
Revised: This is avoided by AGE networks (Ulyanov et al., 2018) and CycleGANs (Zhu et al., 2017a), which combine standard cGAN discriminators with L1 reconstruction loss in the data domain, and bidirectional conditional GANs (Jaiswal et al., 2017), which extend the GAN discriminator to act on the distributions in data and latent space jointly.

Original: SPADE (Park et al., 2019), building upon pix2pix and pix2pixHD (Wang et al., 2018), augments cGANs with additional VAE encoders to shape the latent space such that diversity is ensured.
Revised: SPADE (Park et al., 2019), building upon pix2pix and pix2pixHD (Wang et al., 2018), augments cGANs with additional VAE encoders to shape the latent space such that diversity is ensured.

Original: Instead of enforcing bijectivity through cycle losses, invertible neural networks are bidirectional by design, since encoder and generator are realized by forward and backward processing within a single bijective model.
Revised: Instead of enforcing bijectivity through cycle losses, invertible neural networks are bidirectional by design, since encoder and generator are realized by forward and backward processing within a single bijective model.

Original: We focus on architectures whose forward and backward pass require the same computational effort.
Revised: We focus on architectures whose forward and backward pass require the same computational effort.

Original: The coupling layer designs pioneered by NICE (Dinh et al., 2014) and RealNVP (Dinh et al., 2016) emerged as very powerful and ﬂexible model families under this restriction.
Revised: The coupling layer designs pioneered by NICE (Dinh et al., 2014) and RealNVP (Dinh et al., 2016) emerged as very powerful and ﬂexible model families under this restriction.

Original: Using additive coupling layers, i-RevNets (Jacobsen et al., 2018b) demonstrated that the lack of information reduction from data space to latent space does not cause overﬁtting.
Revised: Using additive coupling layers, i-RevNets (Jacobsen et al., 2018b) demonstrated that the lack of information reduction from data space to latent space does not cause overﬁtting.

Original: The Glow architecture (Kingma & Dhariwal, 2018) combines afﬁne coupling layers with invertible 1x1 convolutions and achieves impressive attribute manipulations (e.g. age, hair color) in generated faces images.
Revised: The Glow architecture (Kingma & Dhariwal, 2018) combines afﬁne coupling layers with invertible 1x1 convolutions and achieves impressive attribute manipulations (e.g. age, hair color) in generated faces images.

Original: This approach was recently generalized to video by Kumar et al.
Revised: This approach was recently generalized to video by Kumar et al.

Original: Thanks to tractable Jacobian determinants, the coupling layer architecture enables maximum likelihood training (Dinh et al., 2014; 2016), but experimental comparisons with other training methods are inconclusive so far.
Revised: Thanks to tractable Jacobian determinants, the coupling layer architecture enables maximum likelihood training (Dinh et al., 2014; 2016), but experimental comparisons with other training methods are inconclusive so far.

Original: For instance, Danihelka et al.
Revised: For instance, Danihelka et al.

Original: (2017) found minimization of an adversarial loss to be superior to maximum likelihood training in RealNVPs, Schirrmeister et al.
Revised: (2017) found minimization of an adversarial loss to be superior to maximum likelihood training in RealNVPs, Schirrmeister et al.

Original: (2018) trained i-RevNets in the same manner as adversarial auto-encoders, i.e. with a discriminator acting in latent rather than data space, and Flow-GANs (Grover et al., 2018) performed best using bidirectional training, a combination of maximum likelihood and adversarial loss.
Revised: (2018) trained i-RevNets in the same manner as adversarial auto-encoders, i.e. with a discriminator acting in latent rather than data space, and Flow-GANs (Grover et al., 2018) performed best using bidirectional training, a combination of maximum likelihood and adversarial loss.

Original: On the other hand, maximum likelihood training worked well within Glow (Kingma & Dhariwal, 2018), and i-ResNets (Behrmann et al., 2018) could even be trained with approximated Jacobian determinants.
Revised: On the other hand, maximum likelihood training worked well within Glow (Kingma & Dhariwal, 2018), and i-ResNets (Behrmann et al., 2018) could even be trained with approximated Jacobian determinants.

Original: In this work we reinforce the view that high-quality generative models can be trained by maximum likelihood loss alone.
Revised: In this work we reinforce the view that high-quality generative models can be trained by maximum likelihood loss alone.

Original: To the best of our knowledge, we are the ﬁrst to apply the coupling layer design for conditional generative models, with the exception of Ardizzone et al.
Revised: To the best of our knowledge, we are the ﬁrst to apply the coupling layer design for conditional generative models, with the exception of Ardizzone et al.

Original: (2019), who use it to compute posteriors for (relatively small) inverse problems, but do not consider image generation.
Revised: (2019), who use it to compute posteriors for (relatively small) inverse problems, but do not consider image generation.

Revised: While Liu et al.

Revised: (2019) recently proposed an effective post-hoc conditioning scheme for ﬂow-based generative models, their method requires an additional cGAN to identify latent codes fulﬁlling the speciﬁed condition.

Original: Colorization.
Revised: Colorization.

Original: State-of-the-art regression models for colorization produce visually near-perfect images (Iizuka et al., 2016), but do not account for the ambiguity inherent in this inverse problem.
Revised: State-of-the-art regression models for colorization produce visually near-perfect images (Iizuka et al., 2016), but do not account for the ambiguity inherent in this inverse problem.

Original: To address this, models would ideally deﬁne a conditional distribution of plausible color images for a given grayscale input, instead of just returning a single “best” solution.
Revised: To address this, models would ideally deﬁne a conditional distribution of plausible color images for a given grayscale input, instead of just returning a single “best” solution.

Original: Popular existing approaches for diverse colorization predict per-pixel color histograms from a U-Net (Zhang et al., 2016) or from hypercolumns of an adapted VGG network (Larsson et al., 2016).
Revised: Popular existing approaches for diverse colorization predict per-pixel color histograms from a U-Net (Zhang et al., 2016) or from hypercolumns of an adapted VGG network (Larsson et al., 2016).

Original: However, sampling from these local histograms independently can not lead to a spatially consistent colorization, requiring additional heuristic post-processing steps to avoid artefacts.
Revised: However, sampling from these local histograms independently can not lead to a spatially consistent colorization, requiring additional heuristic post-processing steps to avoid artefacts.

Original: In terms of generative models, both VAEs (Deshpande et al., 2017) and cGANs (Isola et al., 2017;
Revised: In terms of generative models, both VAEs (Deshpande et al., 2017) and cGANs (Isola et al., 2017;

Original: Cao et al., 2017) have been proposed for the task.
Revised: Cao et al., 2017) have been proposed for the task.

Original: However, their solutions do not reach the quality of the regression-based models, and cGANs in particular often lack diversity.
Revised: However, their solutions do not reach the quality of the regression-based models, and cGANs in particular often lack diversity.

Original: To compensate, modiﬁcations and extensions to generative approaches have been developed, such as auto-regressive models (Guadarrama et al., 2017) and CRFs (Royer et al., 2017).
Revised: To compensate, modiﬁcations and extensions to generative approaches have been developed, such as auto-regressive models (Guadarrama et al., 2017) and CRFs (Royer et al., 2017).

Original: However, these methods are computationally very expensive and often unable to scale to realistic image sizes.
Revised: However, these methods are computationally very expensive and often unable to scale to realistic image sizes.

Original: Conceptually closest to our proposed method is the work of (Ulyanov et al., 2018), where an encoder network maps color information to a latent space and a generator network learns the inverse transform, both conditioned on the grayscale image.
Revised: Conceptually closest to our proposed method is the work of (Ulyanov et al., 2018), where an encoder network maps color information to a latent space and a generator network learns the inverse transform, both conditioned on the grayscale image.

Original: Their experiments however are limited to a data set with only cars, and just three latent dimensions, leading to global, but no local diversity.
Revised: Their experiments however are limited to a data set with only cars, and just three latent dimensions, leading to global, but no local diversity.

Original: In contrast to the above, our ﬂow-based cINN generates diverse colorizations in one standard feedforward pass.
Revised: In contrast to the above, our ﬂow-based cINN generates diverse colorizations in one standard feedforward pass.

Original: It models the distribution of all pixels jointly, and allows for meaningful latent space manipulations.
Revised: It models the distribution of all pixels jointly, and allows for meaningful latent space manipulations.

Original:  M ETHOD
Revised:  M ETHOD

Original: Our method is an extension of the afﬁne coupling block architecture established by Dinh et al.
Revised: Our method is an extension of the afﬁne coupling block architecture established by Dinh et al.

Original: There, each network block splits its input u into two parts [ u 1 , u 2 ] and applies afﬁne transformations between them that have strictly upper or lower triangular Jacobians:
Revised: There, each network block splits its input u into two parts [ u 1 , u 2 ] and applies afﬁne transformations between them that have strictly upper or lower triangular Jacobians:

Original: [Equation]
Revised: [Equation]

Original: The outputs [ v 1 , v 2 ] are concatenated again and passed to the next coupling block.
Revised: The outputs [ v 1 , v 2 ] are concatenated again and passed to the next coupling block.

Original: The internal functions s j and t j can be represented by arbitrary neural networks, and are only ever evaluated in the forward direction, even when the coupling block is inverted:
Revised: The internal functions s j and t j can be represented by arbitrary neural networks, and are only ever evaluated in the forward direction, even when the coupling block is inverted:

Original: [Equation]
Revised: [Equation]

Original: As shown by Dinh et al.
Revised: As shown by Dinh et al.

Original: the logarithm of the Jacobian determinant for such a coupling block is simply the sum of s 1 and s 2 over image dimensions.
Revised: the logarithm of the Jacobian determinant for such a coupling block is simply the sum of s 1 and s 2 over image dimensions.

Original: 3. C ONDITIONAL INVERTIBLE TRANSFORMATIONS
Revised: 3. C ONDITIONAL INVERTIBLE TRANSFORMATIONS

Original: We adapt the design of Eqs.
Revised: We adapt the design of Eqs.

Original: (1) and (2) to produce a conditional version of the coupling block.
Revised: (1) and (2) to produce a conditional version of the coupling block.

Original: Because the subnetworks s j and t j are never inverted, we can concatenate conditioning data c to their inputs without losing the invertibility, replacing s 1 ( u 2 ) with s 1 ( u 2 , c ) etc.
Revised: Because the subnetworks s j and t j are never inverted, we can concatenate conditioning data c to their inputs without losing the invertibility, replacing s 1 ( u 2 ) with s 1 ( u 2 , c ) etc.

Original: Our conditional coupling block design is illustrated in Fig.
Revised: Our conditional coupling block design is illustrated in Fig.

Original: CC in uu(cid:12) + v(cid:12) + vout s 1 ts 2 tc
Revised: [Figure]

Original: In general, we will refer to a cINN with network parameters θ as f ( x ; c , θ ) , and the inverse as g ( z ; c , θ ) .
Revised: In general, we will refer to a cINN with network parameters θ as f ( x ; c , θ ) , and the inverse as g ( z ; c , θ ) .

Original: For any ﬁxed condition c , the invertibility is given as
Revised: For any ﬁxed condition c , the invertibility is given as

Original: [Equation]
Revised: [Equation]

Original: 3. M AXIMUM LIKELIHOOD TRAINING OF C INN S
Revised: 3. M AXIMUM LIKELIHOOD TRAINING OF C INN S

Original: By prescribing a probability distribution p Z ( z ) on latent space Z , the model f assigns any input x a probability, dependent on both the network parameters θ and the conditioning c , through the change-of-variables formula: 
Revised: By prescribing a probability distribution p Z ( z ) on latent space Z , the model f assigns any input x a probability, dependent on both the network parameters θ and the conditioning c , through the change-of-variables formula: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Here, we use the Jacobian matrix ∂f/∂ x .
Revised: Here, we use the Jacobian matrix w.r.t. x , ∂f ( x ; c ) /∂ x .

Original: We will denote the determinant of the Jacobian, evaluated at some training sample x i , as J i ≡ det (cid:0) ∂f/∂ x | x i (cid:1) .
Revised: We will denote the determinant of the Jacobian, evaluated at some training sample x i , as J i ≡ det (cid:0) ∂f/∂ x | x i (cid:1) .

Original: Bayes’ theorem gives us the posterior over model parameters as p ( θ ; x , c ) ∝ p X ( x ; c , θ ) · p θ ( θ ) .
Revised: Bayes’ theorem gives us the posterior over model parameters as p ( θ ; x , c ) ∝ p X ( x ; c , θ ) · p θ ( θ ) .

Original: Our goal is to ﬁnd network parameters that maximize its logarithm, i.e. we minimize the loss
Revised: Our goal is to ﬁnd network parameters that maximize its logarithm, i.e. we minimize the loss

Original: [Equation]
Revised: [Equation]

Original: which is the same as in classical Bayesian model ﬁtting.
Revised: which is the same as in classical Bayesian model ﬁtting.

Original: Inserting Eq.
Revised: Inserting Eq.

Original: (4) with a standard normal distribution for p Z ( z ) , as well as a Gaussian prior on the weights θ with 1 / 2 σ 2 θ ≡ τ , we obtain
Revised: (4) with a standard normal distribution for p Z ( z ) , as well as a Gaussian prior on the weights θ with 1 / 2 σ 2 θ ≡ τ , we obtain

Original: [Equation]
Revised: [Equation]

Original: The latter term represents L2 weight regularization, while the former is the maximum likelihood loss .
Revised: The latter term represents L2 weight regularization, while the former is the maximum likelihood loss .

Original: Training a network with this loss yields an estimate of the maximum likelihood network parameters ˆ θ ML .
Revised: Training a network with this loss yields an estimate of the maximum likelihood network parameters ˆ θ ML .

Original: From there, we can perform conditional generation for a ﬁxed c by sampling z and using the inverted network g : x gen = g ( z ; c , ˆ θ ML ) , with z ∼ p
Revised: From there, we can perform conditional generation for a ﬁxed c by sampling z and using the inverted network g : x gen = g ( z ; c , ˆ θ ML ) , with z ∼ p

Original: Z
Revised: Z

Original: ( z ) .
Revised: ( z ) .

Original: Training with the maximum likelihood method makes it virtually impossible for mode collapse to occur: If any mode in the training set has low probability under the current guess p X ( x ; c , θ ) , the corresponding latent vectors will lie far outside the normal distribution p Z and receive big loss from the ﬁrst L2-term in Eq.
Revised: Training with the maximum likelihood method makes it virtually impossible for mode collapse to occur: If any mode in the training set has low probability under the current guess p X ( x ; c , θ ) , the corresponding latent vectors will lie far outside the normal distribution p Z and receive big loss from the ﬁrst L2-term in Eq.

Original: In contrast, the discriminator of a GAN only supplies a weak signal, proportional to the mode’s relative frequency in the training data, so that the generator is not penalized much for ignoring a mode completely.
Revised: In contrast, the discriminator of a GAN only supplies a weak signal, proportional to the mode’s relative frequency in the training data, so that the generator is not penalized much for ignoring a mode completely.

Original: 3. C ONDITIONING NETWORK
Revised: 3. C ONDITIONING NETWORK

Original: In complex settings, we expect that higher-level features of c need to be extracted for the conditioning to be effective, e.g. global semantic information from an image as in Section 4.2.
Revised: In complex settings, we expect that higher-level features of c need to be extracted for the conditioning to be effective, e.g. global semantic information from an image as in Section 4.2.

Original: In such cases, feeding the condition c directly into the cINN would place an unreasonable burden on the s and t networks, as higher-level features would have to be re-learned in each coupling block.
Revised: In such cases, feeding the condition c directly into the cINN would place an unreasonable burden on the s and t networks, as higher-level features would have to be re-learned in each coupling block.

Original: To address this issue, we introduce an additional feed-forward conditioning network h , which transforms the condition c to some intermediate representation ˜ c
Revised: To address this issue, we introduce an additional feed-forward conditioning network h , which transforms the condition c to some intermediate representation ˜ c

Original: = h ( c ) , and replace c i in Eq.
Revised: = h ( c ) , and replace c i in Eq.

Original: (6) with ˜ c
Revised: (6) with ˜ c

Original: i = h
Revised: i = h

Original: ( c i ) .
Revised: ( c i ) .

Original: The network h can be pretrained, e.g. by using features from a VGG architecture trained for image classiﬁcation.
Revised: The network h can be pretrained, e.g. by using features from a VGG architecture trained for image classiﬁcation.

Original: Alternatively or additionally, h can be trained jointly with the cINN by propagating gradients from the maximum likelihood loss through the conditioning ˜ c .
Revised: Alternatively or additionally, h can be trained jointly with the cINN by propagating gradients from the maximum likelihood loss through the conditioning ˜ c .

Original: In this case, the conditioning network will learn to extract features which are particularly useful for embedding the cINN inputs x into latent variables z .
Revised: In this case, the conditioning network will learn to extract features which are particularly useful for embedding the cINN inputs x into latent variables z .

Original: 3. I MPORTANT DETAILS
Revised: 3. I MPORTANT DETAILS

Original: For cINNs to match the performance of well-established architectures for conditional generation, we introduce a number of minor modiﬁcations and adjustments to the architecture and training procedure.
Revised: For cINNs to match the performance of well-established architectures for conditional generation, we introduce a number of minor modiﬁcations and adjustments to the architecture and training procedure.

Original: With these adaptions, our training setup is very stable and converges every time.
Revised: With these adaptions, our training setup is very stable and converges every time.

Original: Noise as data augmentation.
Revised: Noise as data augmentation.

Original: We add a small amount of noise to the inputs x as part of the standard data augmentation.
Revised: We add a small amount of noise to the inputs x as part of the standard data augmentation.

Original: This helps to smooth out quantization artifacts in the input, and prevents sparse gradients when large parts of the image are completely ﬂat (as e.g. in MNIST).
Revised: This helps to smooth out quantization artifacts in the input, and prevents sparse gradients when large parts of the image are completely ﬂat (as e.g. in MNIST).

Original: Soft clamping of scale coefﬁcients.
Revised: Soft clamping of scale coefﬁcients.

Original: We apply an additional nonlinear function to the scale coefﬁcients s , of the form s clamp =
Revised: We apply an additional nonlinear function to the scale coefﬁcients s , of the form s clamp =

Original: (2 α/π )arctan( s/α ) , which yields s clamp
Revised: (2 α/π )arctan( s/α ) , which yields s clamp

Original: ≈ s for | s | (cid:28) α and s clamp
Revised: ≈ s for | s | (cid:28) α and s clamp

Original: ≈ ±
Revised: ≈ ±

Original: α for | s | (cid:29) α .
Revised: α for | s | (cid:29) α .

Original: This prevents any instabilities stemming from exploding magnitude of the exponential exp( s clamp ) .
Revised: This prevents any instabilities stemming from exploding magnitude of the exponential exp( s clamp ) .

Original: We ﬁnd α = 1 .
Revised: We ﬁnd α = 1 .

Original: 5 work well.
Revised: 5 work well.

Revised: A similar scheme is used in the realNVP architecture (Dinh et al., 2016), whereby α is a learned vector, instead of a ﬁxed hyperparameter.

Revised: We ﬁnd that a ﬁxed α does not limit performance, while completely ruling out problems from exploding or vanishing scaling factors.

Original: Initialization.
Revised: Initialization.

Original: Heuristically, we ﬁnd that Xavier initialization (Glorot & Bengio, 2010) leads to stable training from the start.
Revised: Heuristically, we ﬁnd that Xavier initialization (Glorot & Bengio, 2010) leads to stable training from the start.

Original: We experienced training instability when initial parameter values were too high.
Revised: We experienced training instability when initial parameter values were too high.

Original: Similar to Kingma & Dhariwal (2018), we also initialize the last convolution in all s and t subnetworks to zero, so training starts from an identity transform.
Revised: Similar to Kingma & Dhariwal (2018), we also initialize the last convolution in all s and t subnetworks to zero, so training starts from an identity transform.

Original: Soft channel permutations.
Revised: Soft channel permutations.

Original: We use random orthogonal matrices to mix the information between the channels.
Revised: We use random orthogonal matrices to mix the information between the channels.

Original: This allows for more interaction between the two information streams u 1 , u 2 in the coupling blocks.
Revised: This allows for more interaction between the two information streams u 1 , u 2 in the coupling blocks.

Original: A similar technique was used by Kingma & Dhariwal (2018), but our matrices stay ﬁxed throughout training and are guaranteed to be cheaply invertible.
Revised: A similar technique was used by Kingma & Dhariwal (2018), but our matrices stay ﬁxed throughout training and are guaranteed to be cheaply invertible.

Original: Haar wavelet downsampling.
Revised: Haar wavelet downsampling.

Original: All prior INN architectures use checkerboard patterns for reshaping to lower spatial resolutions.
Revised: All prior INN architectures use checkerboard patterns for reshaping to lower spatial resolutions.

Original: We ﬁnd it helpful to instead perform downsampling with Haar wavelets (Haar, 1910), which essentially decompose images into an average pooling channel as well as vertical, horizontal and diagonal derivatives, see Fig. The three derivative channels contain high resolution information which we can split off early, transforming only the remaining information further in later stages of the cINN. This also contributes tomixing the variables between layers, complementing the soft permutations.
Revised: However, this possibly results in unnatural inputs to the following layers, where previously neighboring pixels are suddenly treated independently as distinct channels. We argue, supported by our ablation experiments, that a deliberately chosen spatial decomposition can be more natural, and more closely resemble the effect of pooling operations in feed-forward networks.

Original: Similarly, Jacobsen et al. (2018a) use a discrete cosine transform as a ﬁnal transformation in their INN, to replace global average pooling.
Revised: For these reasons, we perform downsampling with Haar wavelets (Haar, 1910), which essentially

Original: 13c × 2 ×= (cid:32) 1212average , 12 (cid:57)12 (cid:57)horizontal , 12 12 (cid:57) 12 (cid:57)vertical , 12 (cid:57) 12 (cid:57) 12diagonal (cid:33) · a h v d 4 · c × 1 ×
Revised: 13c × 2 ×= (cid:32) 1212average , 12 (cid:57)12 (cid:57)horizontal , 12(cid:57) 12 (cid:57)vertical , 12 (cid:57)(cid:57) 12diagonal (cid:33) · a h v d 4 · c × 1 ×

Original: 28 ×x
Revised: 28 ×x

Original: CC f u ll yc onn ec t e d
Revised: CC f u ll yc onn ec t e d

Original: CC f u ll yc onn ec t e d ...
Revised: CC f u ll yc onn ec t e d ...

Original: CC f u ll yc onn ec t e d 784 (28 × 28) 
Revised: CC f u ll yc onn ec t e d 784 (28 × 28) z

Original: z 24 blocks (cid:2) 0 0 0 1 0 0 0 0 0 0 (cid:3) c  as one-hot vector
Revised: 24blocks (cid:2) 0 0 0 1 0 0 0 0 0 0 (cid:3) c asone-hotvector decompose images into a 2 × 2 average pooling channel as well as vertical, horizontal and diagonal derivatives, see Fig. This also motivates which channels should be split off to directly become latent variables, namely the derivative channels with high-resolution information, and which should be transformed further in later stages of the cINN, i.e. the pooling channels (see Sec. 4.2). The wavelet transform also contributes to mixing the variables between layers, complementing the soft permutations. Similarly, Jacobsen et al.

Revised: (2018a) use a global discrete cosine transform as a ﬁnal transformation in their INN, to replace global average pooling.

Original: Discussion.
Revised: Discussion.

Original: The soft clamping, along with the Haar wavelet downsampling are critical for training stability at sufﬁciently high learning rates.
Revised: The soft clamping, along with the Haar wavelet downsampling are critical for training stability at sufﬁciently high learning rates.

Original: With them, the network is exceptionally stable, and converges to a good optimum every time.
Revised: With them, the network is exceptionally stable, and converges to a good optimum every time.

Original: The noise augmentation and initialization additionally increase the convergence speed.
Revised: The noise augmentation and initialization additionally increase the convergence speed.

Original: The permutations are necessary to achieve sufﬁcient expressive power and high quality results, but do not affect the training stability or speed.
Revised: The permutations are necessary to achieve sufﬁcient expressive power and high quality results, but do not affect the training stability or speed.

Original: Ablation curves for these improvements are given in the appendix.
Revised: Ablation curves for these improvements are given in the appendix.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: We present results and explore the latent space of our models for two conditional image generation tasks:
Revised: We present results and explore the latent space of our models for two conditional image generation tasks:

Original: MNIST digit generation and image colorization.
Revised: MNIST digit generation and image colorization.

Original: 4. C LASS - CONDITIONAL GENERATION FOR MNIST
Revised: 4. C LASS - CONDITIONAL GENERATION FOR MNIST

Original: As a ﬁrst experiment, we perform simple class-conditional generation of MNIST digits.
Revised: As a ﬁrst experiment, we perform simple class-conditional generation of MNIST digits.

Original: We construct a cINN of 24 coupling blocks using fully connected subnetworks s and t , which receive the conditioning directly as a one-hot vector (Fig. 4).
Revised: We construct a cINN of 24 coupling blocks using fully connected subnetworks s and t , which receive the conditioning directly as a one-hot vector (Fig. 4).

Original: No conditioning network h is used.
Revised: No conditioning network h is used.

Original: For data augmentation we only add a small amount of noise to the images ( σ = 0 . 02 ), as described in Section 3.4.
Revised: For data augmentation we only add a small amount of noise to the images ( σ = 0 . 02 ), as described in Section 3.4.

Original: Samples generated by the model are shown in Fig.
Revised: Samples generated by the model are shown in Fig.

Original: We ﬁnd that the cINN learns latent representations that are shared across conditions c .
Revised: We ﬁnd that the cINN learns latent representations that are shared across conditions c .

Original: Keeping the latent vector z ﬁxed while varying c produces different digits in the same style.
Revised: Keeping the latent vector z ﬁxed while varying c produces different digits in the same style.

Original: This property, in conjunction with our network’s invertibility, can directly be used for style transfer, as demonstrated in Fig.
Revised: This property, in conjunction with our network’s invertibility, can directly be used for style transfer, as demonstrated in Fig.

Original: This outcome is not obvious – the trained cINN could also decompose into 10 essentially separate subnetworks, one for each condition.
Revised: This outcome is not obvious – the trained cINN could also decompose into 10 essentially separate subnetworks, one for each condition.

Original: In this case, the latent space of each class would be structured differently, and inter-class transfer of latent vectors would be meaningless.
Revised: In this case, the latent space of each class would be structured differently, and inter-class transfer of latent vectors would be meaningless.

Original: The structure of the latent space is further illustrated in Fig.
Revised: The structure of the latent space is further illustrated in Fig.

Original: 5, where we identify three latent axes with interpretable meanings.
Revised: 5, where we identify three latent axes with interpretable meanings.

Original: Note that while the latent space is learned without supervision, we found the axes in a semi-automatic fashion: We perform PCA on the latent vectors of the test set, without the noise augmentation, and manually identify meaningful directions  in the subspace of the ﬁrst four principal components.
Revised: Note that while the latent space is learned without supervision, we found these axes in a semi-automatic fashion: We perform PCA on the latent vectors of the test set, without the noise augmentation. Since disentanglement is only unique up to a rotation in latent space, the directions corresponding to human-interpretable properties do not necessarily align with the latent axes. We therefore manually ﬁnd a set of orthogonal vectors in the subspace of the ﬁrst four PCA components which represent such properties. [Figure]

Revised: [Figure]

Original: 4. D IVERSE IMAGE COLORIZATION
Revised: 4. D IVERSE IMAGE COLORIZATION

Original: For a more challenging task, we turn to colorization of natural images.
Revised: For a more challenging task [Figure] we turn to colorization of natural images.

Original: The common approach for this task is to represent images in Lab color space and generate color channels a , b by a model conditioned on the luminance channel L .
Revised: The common approach for this task is to represent images in Lab color space and generate color channels a , b by a model conditioned on the luminance channel L .

Original: [Figure]
Revised: We train on the ImageNet dataset (Russakovsky et al., 2015), again adding low noise to the a , b channels ( σ = 0 . 05 ).

Original: [Figure] do not require as much resolution as the luminance channel, we condition on 256 × 256 pixel grayscale images, but generate 64 × 64 pixel color information.
Revised: As the color channels do not require as much resolution as the luminance channel, we condition on 256 × 256 pixel grayscale images, but generate 64 × 64 pixel color information.

Original: This is in accordance with the majority of existing colorization methods.
Revised: This is in accordance with the majority of existing colorization methods.

Original: As with most generative INN architectures, we do not keep the resolution and channels ﬁxed throughout the network, for the sake of computational cost.
Revised: As with most generative INN architectures, we do not keep the resolution and channels ﬁxed throughout the network, for the sake of computational cost.

Original: Instead, we use 4 resolution stages, as illustrated in Fig. 8.
Revised: Instead, we use 4 resolution stages, as illustrated in Fig. 8.

Original: At each stage, the data is reshaped to a lower resolution and more channels, after which a fraction of the channels are split off as one part of the latent code.
Revised: At each stage, the data is reshaped to a lower resolution and more channels, after which a fraction of the channels are split off as one part of the latent code.

Original: As the high resolution stages have a smaller receptive ﬁeld and less expressive power, the corresponding parts of the latent vector encode local structures and noise.
Revised: As the high resolution stages have a smaller receptive ﬁeld and less expressive power, the corresponding parts of the latent vector encode local structures and noise.

Original: More global information is passed on to the lower resolution sections of the cINN.
Revised: More global information is passed on to the lower resolution sections of the cINN.

Original: For the conditioning network h , we start with the same VGG-like architecture from Zhang et al.
Revised: For the conditioning network h , we start with the same VGG-like architecture from Zhang et al.

Original: (2016) and pretrain on the colorization task.
Revised: (2016) and pretrain on the colorization task.

Original: By cutting off the network before the second-to-last convolution, we extract 256 feature maps of size 64 × 64 from the grayscale image L .
Revised: By cutting off the network before the second-to-last convolution, we extract 256 feature maps of size 64 × 64 from the grayscale image L .

Original: We then add independent heads on top of this for each conditional coupling block in the cINN, indicated by small hexagons in Fig. 8.
Revised: We then add independent heads on top of this for each conditional coupling block in the cINN, indicated by small hexagons in Fig. 8.

Original: Thus each coupling block k receives its own specialized conditioning ˜ c ( k ) i
Revised: Thus each coupling block k receives its own specialized conditioning ˜ c ( k ) i

Original: = h k (cid:0) h
Revised: = h k (cid:0) h

Original: ( c i ) (cid:1) .
Revised: ( c i ) (cid:1) .

Original: Each head consists of up to ﬁve strided convolutions, depending on its required output resolution, and a batch normalization layer.
Revised: Each head consists of up to ﬁve strided convolutions, depending on its required output resolution, and a batch normalization layer.

Original: The ablation study in Fig.
Revised: The ablation study in Fig.

Original: 14 conﬁrms that the conditioning network is necessary to capture semantic information.
Revised: 14 conﬁrms that the conditioning network is necessary to capture semantic information.

Original: We initially train the cINN and the h k , keeping the parameters of the conditioning network h ﬁxed, for 30 000 iterations.
Revised: We initially train the cINN and the h k , keeping the parameters of the conditioning network h ﬁxed, for 30 000 iterations.

Original: After this, we train both jointly until convergence, for 3 days on 3 Nvidia
Revised: After this, we train both jointly until convergence, for 3 days on 3 Nvidia

Revised: [Table]

Revised: For each of 5k ImageNet validation images, we compare the best pixel-wise MSE of 8 generated colorization samples, the pixel-wise variance between the 8 samples as an approximation of the diversity, the Fréchet Inception Distance (Heusel et al., 2017) as a measure of realism, and the topaccuracy of ImageNet classiﬁcation performed on the colorized images, to check if semantic content is preserved by the colorization.

Revised: ∗ Note: It was not possible to process all test images with PIC, due to the long inference times.

Revised: Only 768 images are used.

Revised: Full results will be included in the camera ready version.

Original: GTX1080 GPUs.
Revised: GTX1080 GPUs.

Original: The Adam optimizer is essential for fast convergence, and we lower the learning rate when the maximum likelihood loss levels off.
Revised: The Adam optimizer is essential for fast convergence, and we lower the learning rate when the maximum likelihood loss levels off.

Original: At inference time, we use joint bilateral upsampling (Kopf et al., 2007) to match the resolution of the generated color channels ˆ a , ˆ b to that of the luminance channel L .
Revised: At inference time, we use joint bilateral upsampling (Kopf et al., 2007) to match the resolution of the generated color channels ˆ a , ˆ b to that of the luminance channel L .

Original: This produces visually slightly more pleasing edges than bicubic upsampling, but has little to no impact on the results.
Revised: This produces visually slightly more pleasing edges than bicubic upsampling, but has little to no impact on the results.

Original: It was not used in the quantitative results table, to ensure an unbiased comparison.
Revised: It was not used in the quantitative results table, to ensure an unbiased comparison.

Original: Latent space interpolations and color transfer are shown in Figs.
Revised: Latent space interpolations and color transfer are shown in Figs.

Original: 9 and 10, with more experiments in the appendix.
Revised: 9 and 10, with more experiments in the appendix.

Original: In Table 1, a quantitative comparison to existing methods is given.
Revised: In Table 1, a quantitative comparison to existing methods is given.

Revised: While on par with

Original: The cINN clearly has the best sample diversity, as summarized by the variance and best-of-8 accuracy.
Revised: VAE-MDN in other metrics, the cINN clearly has the best sample diversity, as summarized by the variance and best-of-8 accuracy.

Original: The cGAN completely ignores the latent code, and relies only on the condition. As a result, we do not observe any measurable diversity, in line with results from Isola et al.
Revised: The cGAN completely ignores the latent code and relies only on the condition – there is no diversity, in line with results from Isola et al.

Original: In terms of FID score, the cGAN performs best, although its results do not appear more realistic to the human eye, cf.
Revised: In terms of FID score, the cGAN performs best, although its results do not appear more realistic to the human eye, cf.

Original: Fig.
Revised: Fig.

Original: This may be due to the fact that FID is sensitive to outliers, which are unavoidable for a truly diverse method (see Fig. 12), or because the discriminator loss implicitly optimizes for the similarity of deep
Revised: This may be due to the fact that FID is sensitive to outliers, which are unavoidable for a truly diverse method (see Fig. 12), or because the discriminator loss implicitly optimizes for the similarity of deep

Original: CNN activations.
Revised: CNN activations.

Original: The VGG classiﬁcation accuracy of colorized images is decreased for all generative methods equally, because occasional outliers may lead to misclassiﬁcation.
Revised: The VGG classiﬁcation accuracy of colorized images is decreased for all generative methods equally, because occasional outliers may lead to misclassiﬁcation.

Original: Because the diverse COLORGAN by Cao et al.
Revised: Because the diverse COLORGAN by Cao et al.

Original: (2017) was exclusively trained on the LSUN bedrooms dataset (Yu et al., 2015) at small resolution, and would not converge on ImageNet, we compare this separately and provide results in the appendix.
Revised: (2017) was exclusively trained on the LSUN bedrooms dataset (Yu et al., 2015) at small resolution, and would not converge on ImageNet, we compare this separately and provide results in the appendix.

Original: In this simple case, the performance of the cINN and
Revised: In this simple case, the performance of the cINN and

Original: COLORGAN is almost equal in all apsects, with much shorter training time for the cINN.
Revised: COLORGAN is almost equal in all apsects, with much shorter training time for the cINN.

Original:  C ONCLUSION AND O UTLOOK
Revised:  C ONCLUSION AND O UTLOOK

Original: We have proposed a conditional invertible neural network architecture which enables guided generation of diverse images with high realism.
Revised: We have proposed a conditional invertible neural network architecture which enables guided generation of diverse images with high realism.

Original: For image colorization, we believe that even better results can be achieved when employing the latest tricks from large-scale GAN frameworks.
Revised: For image colorization, we believe that even better results can be achieved when employing the latest tricks from large-scale GAN frameworks.

Original: Especially the non-invertible nature of the conditioning network makes cINNs a suitable method for other computer vision tasks such as diverse semantic segmentation.
Revised: Especially the non-invertible nature of the conditioning network makes cINNs a suitable method for other computer vision tasks such as diverse semantic segmentation.

Original: [Figure]
Revised: [Figure]

Original: [Equation]

Original: Vector z ∗ is “typical” in the sense that (cid:107) z ∗ (cid:107) 2 = E (cid:2) (cid:107) z (cid:107) 2 (cid:3) , and results in natural colors.

Original: As we move closer to the center of the latent space ( (cid:107) z (cid:107) < (cid:107) z ∗ (cid:107) ), regions with ambiguous colors become desaturated, while less ambiguous regions (e.g. sky, vegetation) revert to their prototypical colors.

Original: In the opposite direction ( (cid:107) z (cid:107) >

Original: (cid:107) z ∗ (cid:107) ), colors are enhanced to the point of oversaturation.

Original: [Table]

Original: [Figure]
Revised: [Figure]

Original: A A DDITIONAL E XPERIMENTS A. A BLATION OF TRAINING IMPROVEMENTS
Revised: A A DDITIONAL E XPERIMENTS A. A BLATION OF TRAINING IMPROVEMENTS

Original: To demonstrate the improved stability and training speed through the improvements from Sec. 3. we perform ablations, see Fig. 
Revised: To demonstrate the improved stability and training speed through the improvements from Sec. 3.4, we perform ablations, see Fig. 15.

Original: The ablations for colorization were performed for the LSUN bedrooms task, due to training speed.
Revised: The ablations for colorization were performed for the LSUN bedrooms task, due to training speed.

Original: We ﬁnd that for stable training at Adam learning rates of 10 − 3 , the clamping and Haar wavelet downsampling are strictly necessary.
Revised: We ﬁnd that for stable training at Adam learning rates of 10 − 3 , the clamping and Haar wavelet downsampling are strictly necessary.

Original: Without these, the network has to be trained with much lower learning rates and more careful and specialized initialization, as used e.g. in Kingma & Dhariwal (2018).
Revised: Without these, the network has to be trained with much lower learning rates and more careful and specialized initialization, as used e.g. in Kingma & Dhariwal (2018).

Original: Beyond this, the noise augmentation and permutations lead to the largest improvement in ﬁnal result.
Revised: Beyond this, the noise augmentation and permutations lead to the largest improvement in ﬁnal result.

Original: The effect of the noise is more pronounced for MNIST, as large parts of the image are completely black otherwise.
Revised: The effect of the noise is more pronounced for MNIST, as large parts of the image are completely black otherwise.

Original: For natural images, dequantization of the data is likely to be the main advantage of the added noise.
Revised: For natural images, dequantization of the data is likely to be the main advantage of the added noise.

Original: Note however, that the training curves of the models with and without noise augmentation is not directly comparable, as the loss differs an additional summand ≈ log( σ aug . ) .
Revised: Note however, that the training curves of the models with and without noise augmentation is not directly comparable, as the loss differs an additional summand ≈ log( σ aug . ) .

Original: The effect on the training speed and stability is clearly visible regardless.
Revised: The effect on the training speed and stability is clearly visible regardless.

Original: The initialization only improves the ﬁnal result by a small margin, but also converges noticeably faster. 
Revised: The initialization only improves the ﬁnal result by a small margin, but also converges noticeably faster. [Figure] [Table]

Original: [Figure]
Revised: [Figure]

Original: A. LSUN B EDROOMS
Revised: A. LSUN B EDROOMS

Original: To provide a simpler model for more in-depth experiments and ablations, we additionally train a cINN for colorization on the LSUN bedrooms dataset Yu et al.
Revised: To provide a simpler model for more in-depth experiments and ablations, we additionally train a cINN for colorization on the LSUN bedrooms dataset Yu et al.

Original: We use a smaller model than for ImageNet, and train the conditioning network jointly from scratch, without pretraining.
Revised: We use a smaller model than for ImageNet, and train the conditioning network jointly from scratch, without pretraining.

Original: Both the conditioning input, as well as the generated color channels have a resolution of 64 × 64 pixels.
Revised: Both the conditioning input, as well as the generated color channels have a resolution of 64 × 64 pixels.

Original: The entire model trains in under 4 hours on a single GTX 1080Ti GPU.
Revised: The entire model trains in under 4 hours on a single GTX 1080Ti GPU.

Original: To our knowledge, the only diversity-enforcing cGAN architecture previously used for colorization is the colorGAN Cao et al.
Revised: To our knowledge, the only diversity-enforcing cGAN architecture previously used for colorization is the colorGAN Cao et al.

Original: (2017), which is also trained exclusively on the bedrooms dataset.
Revised: (2017), which is also trained exclusively on the bedrooms dataset.

Original: Training the colorGAN for comparison, we ﬁnd it requires over 24 hours to converge stably, after multiple restarts.
Revised: Training the colorGAN for comparison, we ﬁnd it requires over 24 hours to converge stably, after multiple restarts.

Original: The results are generally worse than those of the cINN, as shown in Fig.
Revised: The results are generally worse than those of the cINN, as shown in Fig.

Original: While the resulting pixel-wise color variance is slightly higher for the colorGAN, it is not clear whether this captures the true variance, or whether it is due to unrealistically colorful outputs, such as in the second row in Fig.
Revised: While the resulting pixel-wise color variance is slightly higher for the colorGAN, it is not clear whether this captures the true variance, or whether it is due to unrealistically colorful outputs, such as in the second row in Fig.

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: B A DDITIONAL F IGURES 
Revised: B A DDITIONAL F IGURES B. C OLORIZATION – T RANSFER

Revised: The following ﬁgure contains an additional example of the color transfer, shown in Fig.

Original: [Figure]
Revised: 9 in the paper.

Original:  C 
Revised: B. C OLORIZATION – I NTERPOLATIONS

Original: [Figure]

Original: In the following, we show 2-dimensional interpolations in latent space.
Revised: In the following, we show 2-dimensional interpolations in latent space.

Original: Two random latent vectors z (1) , z (2) are linearly combined:
Revised: Two random latent vectors z (1) , z (2) are linearly combined:

Original: [Equation]
Revised: [Equation]

Original: with varying a 1 , a 2 ∈ [ − 0 .
Revised: with varying a 1 , a 2 ∈ [ − 0 .

Original: 9] across each axis of a grid.
Revised: 9] across each axis of a grid.

Original: The center image has z ∗ = 0 .
Revised: The center image has z ∗ = 0 .

Original: Note that the images in the corners have a larger magnitude than trained for, (cid:107) z ∗ (cid:107) 2 ≈ 1 .
Revised: Note that the images in the corners have a larger magnitude than trained for, (cid:107) z ∗ (cid:107) 2 ≈ 1 .

Original: 3 E (cid:2) (cid:107) z (cid:107) 2 (cid:3) , leading to some oversaturation artifacts, as in Fig.
Revised: 3 E (cid:2) (cid:107) z (cid:107) 2 (cid:3) , leading to some oversaturation artifacts, as in Fig.

Original: 12 of the main paper.
Revised: 12 of the main paper.

Revised: [Figure]

Original: B. C OLORIZATION – A DDITIONAL EXAMPLES
Revised: B. C OLORIZATION – A DDITIONAL EXAMPLES

Original: On the following pages, we provide some additional colorized images, as well as comparisons to alternative methods.
Revised: On the following pages, we provide some additional colorized images, as well as comparisons to alternative methods.

Original: All images are taken from the ImageNet 2012 validation set, and all methods were trained on ImageNet 2012.
Revised: All images are taken from the ImageNet 2012 validation set, and all methods were trained on ImageNet 2012.

Original: As we do not observe any signiﬁcant diversity for the cGAN, we only provide a single sample.
Revised: As we do not observe any signiﬁcant diversity for the cGAN, we only provide a single sample.

Original: B.2. G ENERAL EXAMPLES
Revised: B.3. G ENERAL EXAMPLES

Original: [Figure]
Revised: [Figure]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table]
Revised: [Table]

Original: B.2. H UMANS
Revised: B.3. H UMANS

Original: We ﬁnd that the cINN often has difﬁculties generating convincing skin colors, as shown below.
Revised: We ﬁnd that the cINN often has difﬁculties generating convincing skin colors, as shown below.

Original: Clothing is colored in diverse ways, but not always with the correct connectivity and consistency.
Revised: Clothing is colored in diverse ways, but not always with the correct connectivity and consistency.

Original: [Figure]
Revised: [Figure]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: B.2. L ACKING CONSISTENCY
Revised: B.3. L ACKING CONSISTENCY

Original: The following failure cases exhibit a lack in consistency, in occluded objects, multi-part objects, or reﬂections.
Revised: The following failure cases exhibit a lack in consistency, in occluded objects, multi-part objects, or reﬂections.

Original: [Table]
Revised: [Table]

Original: B.2. C OLOR IGNORES SEMANTIC CONTENT
Revised: B.3. C OLOR IGNORES SEMANTIC CONTENT

Original: In the following examples, the semantic content of the image was not recognized, and the generated colors are clearly incorrect.
Revised: In the following examples, the semantic content of the image was not recognized, and the generated colors are clearly incorrect.

Original: [Figure]
Revised: [Figure]

Original: [Table] 
Revised: [Table] [Figure] [Table] [Figure]

Revised: [Table]

Original: [Figure]
Revised: [Figure]

Revised: CNN

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table] 
Revised: [Table] [Figure] [Table] [Figure]

Original: [Table] 
Revised: [Table] [Figure] [Table] [Table]

Original: [Table]
Revised: [Table]

Original: B.2. O UTRIGHT FAILURES
Revised: B.3. O UTRIGHT FAILURES

Original: For the following images, the cINN fails completely, and generates colors with seemingly little or no connection to the grayscale image.
Revised: For the following images, the cINN fails completely, and generates colors with seemingly little or no connection to the grayscale image.

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Table] [Table] 
Revised: [Table] [Table] [Table]