Original: P OWERS OF LAYERS FOR IMAGE - TO - IMAGE TRANSLATION
Revised: P OWERS OF LAYERS FOR IMAGE - TO - IMAGE TRANSLATION

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: We propose a simple architecture to address unpaired image-to-image translation tasks: style or class transfer, denoising, deblurring, deblocking, etc.
Revised: We propose a simple architecture to address unpaired image-to-image translation tasks: style or class transfer, denoising, deblurring, deblocking, etc.

Original: We start from an image autoencoder architecture with ﬁxed weights.
Revised: We start from an image autoencoder architecture with ﬁxed weights.

Original: For each task we learn a residual block operating in the latent space, which is applied iteratively until the target domain is reached.
Revised: For each task we learn a residual block operating in the latent space, which is applied iteratively until the target domain is reached.

Original: A speciﬁc training schedule is required to alleviate the exponentiation effect of the iterations.
Revised: A speciﬁc training schedule is required to alleviate the exponentiation effect of the iterations.

Original: At test time, it offers several advantages: the number of weight parameters is limited and the strength of the transformation can be modulated simply with the number of iterations.
Revised: At test time, it offers several advantages: the number of weight parameters is limited and the strength of the transformation can be modulated simply with the number of iterations.

Original: This is useful, for instance, when the type or amount of noise to suppress is not known in advance.
Revised: This is useful, for instance, when the type or amount of noise to suppress is not known in advance.

Original: Experimentally, we show that the performance of our model is comparable or better than CycleGAN and Nice-GAN with fewer parameters.
Revised: Experimentally, we show that the performance of our model is comparable or better than CycleGAN and Nice-GAN with fewer parameters.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Neural networks deﬁne arbitrarily complex functions involved in discriminative or generative tasks by stacking layers, as supported by the universal approximation theorem (Hornik et al., 1989; Montúfar, 2014).
Revised: Neural networks deﬁne arbitrarily complex functions involved in discriminative or generative tasks by stacking layers, as supported by the universal approximation theorem (Hornik et al., 1989; Montúfar, 2014).

Original: More precisely, the theorem states that stacking a number of basic blocks can approximate any function with arbitrary precision, provided it has enough hidden units, with mild conditions on the non-linear basic blocks.
Revised: More precisely, the theorem states that stacking a number of basic blocks can approximate any function with arbitrary precision, provided it has enough hidden units, with mild conditions on the non-linear basic blocks.

Original: Studies on non-linear complex holomorphic functions involved in escape-time fractals showed that iterating simple non-linear functions can also construct arbitrarily complex landscapes (Barnsley et al., 1988).
Revised: Studies on non-linear complex holomorphic functions involved in escape-time fractals showed that iterating simple non-linear functions can also construct arbitrarily complex landscapes (Barnsley et al., 1988).

Original: These functions are complex in the sense that their iso-surfaces are made arbitrarily large by increasing the number of iterations.
Revised: These functions are complex in the sense that their iso-surfaces are made arbitrarily large by increasing the number of iterations.

Original: Yet there is no control on the actual shape of the resulting function.
Revised: Yet there is no control on the actual shape of the resulting function.

Original: This is why generative fractals remain mathematical curiosities or at best tools to construct intriguing landscapes.
Revised: This is why generative fractals remain mathematical curiosities or at best tools to construct intriguing landscapes.

Original: In this paper, our objective is to combine the expressive power of both constructions, and we study the optimization of a function that iterates a single building block in the latent space of an auto-encoder.
Revised: In this paper, our objective is to combine the expressive power of both constructions, and we study the optimization of a function that iterates a single building block in the latent space of an auto-encoder.

Original: We focus on image translation tasks, that can be trained from either paired or unpaired data.
Revised: We focus on image translation tasks, that can be trained from either paired or unpaired data.

Original: In the paired case, pairs of corresponding input and output images are provided during training.
Revised: In the paired case, pairs of corresponding input and output images are provided during training.

Original: It offers a direct supervision, so the best results are usually obtained with these methods (Chen et al., 2017; Wang et al., 2018; Park et al., 2019).
Revised: It offers a direct supervision, so the best results are usually obtained with these methods (Chen et al., 2017; Wang et al., 2018; Park et al., 2019).

Original: We focus on unpaired translation : only two corpora of images are provided, one for the input domain A and the other for the output domain B .
Revised: We focus on unpaired translation : only two corpora of images are provided, one for the input domain A and the other for the output domain B .

Original: Therefore we do not have access to any parallel data (Conneau et al., 2017), which is a realistic scenario in many applications, e.g., image restoration.
Revised: Therefore we do not have access to any parallel data (Conneau et al., 2017), which is a realistic scenario in many applications, e.g., image restoration.

Original: We train a function f AB : A !
Revised: We train a function f AB : A !

Original: B , such that the output b
Revised: B , such that the output b

Original: ⇤
Revised: ⇤

Original: =
Revised: =

Original: F ( a ) for a 2 A is indiscernible from images of B .
Revised: F ( a ) for a 2 A is indiscernible from images of B .

Original: Our transformation is performed by a single residual block that is composed a variable number of times.
Revised: Our transformation is performed by a single residual block that is composed a variable number of times.

Original: We obtain this compositional property thanks to a progressive learning scheme that ensures that the output is valid for a large range of iterations.
Revised: We obtain this compositional property thanks to a progressive learning scheme that ensures that the output is valid for a large range of iterations.

Original: As a result, we can modulate the strength of the transformation by varying the number of times the transformation is composed.
Revised: As a result, we can modulate the strength of the transformation by varying the number of times the transformation is composed.

Original: This is of particular interest in image translation tasks such as denoising, where the noise level is unknown at training time, and style transfer, where the user may want to select the best rendering.
Revised: This is of particular interest in image translation tasks such as denoising, where the noise level is unknown at training time, and style transfer, where the user may want to select the best rendering.

Original: This “Powers of layers” (PoL) mechanism is illustrated in Figure 1 in the category transfer context (horse to zebra).
Revised: This “Powers of layers” (PoL) mechanism is illustrated in Figure 1 in the category transfer context (horse to zebra).

Original: Our architecture is very simple and only the weights of the residual block differ depending on the task, which makes it suitable to address a large number of tasks with a limited number of parameters.
Revised: Our architecture is very simple and only the weights of the residual block differ depending on the task, which makes it suitable to address a large number of tasks with a limited number of parameters.

Original: This proposal is in sharp contrast with the trend of current state-of-the-art works to specialize the architecture and to increase its complexity and number of parameters (Fu et al., 2019; Viazovetskyi et al., 2020; Choi et al., 2020).
Revised: This proposal is in sharp contrast with the trend of current state-of-the-art works to specialize the architecture and to increase its complexity and number of parameters (Fu et al., 2019; Viazovetskyi et al., 2020; Choi et al., 2020).

Original: Despite its simplicity, our proof of concept exhibits similar or better performance than a vanilla CycleGAN architecture, all things being equal otherwise, for the
Revised: Despite its simplicity, our proof of concept exhibits similar or better performance than a vanilla CycleGAN architecture, all things being equal otherwise, for the

Original: Discriminator
Revised: Discriminator

Original: Encoder
Revised: Encoder

Original: Decoder
Revised: Decoder

Original: Stoppingcriterion
Revised: Stoppingcriterion

Original: C O N V
Revised: C O N V

Original: C O N V N O R M res AB
Revised: C O N V N O R M res AB

Original: R E L U original set of image-to-image translation tasks proposed in their papers, as well as for denoising, deblurring and deblocking.
Revised: R E L U original set of image-to-image translation tasks proposed in their papers, as well as for denoising, deblurring and deblocking.

Original: With signiﬁcantly fewer parameters and a versatile architecture, we report competitive results conﬁrmed by objective and psycho-visual metrics, illustrated by visualizations.
Revised: With signiﬁcantly fewer parameters and a versatile architecture, we report competitive results conﬁrmed by objective and psycho-visual metrics, illustrated by visualizations.

Original: We will provide the implementation for the sake of reproducibility.
Revised: We will provide the implementation for the sake of reproducibility.

Original:  R ELATED WORK
Revised:  R ELATED WORK

Original: Generative adversarial networks (GANs) (Goodfellow et al., 2014) is a framework where two networks, a generator and a discriminator, are learned together in a zero-sum game fashion.
Revised: Generative adversarial networks (GANs) (Goodfellow et al., 2014) is a framework where two networks, a generator and a discriminator, are learned together in a zero-sum game fashion.

Original: The generator learns to produce more and more realistic images w.r.t.
Revised: The generator learns to produce more and more realistic images w.r.t.

Original: the training dataset.
Revised: the training dataset.

Original: The discriminator learns to separate between real data and increasingly realistic generated images.
Revised: The discriminator learns to separate between real data and increasingly realistic generated images.

Original: GANs are used in many tasks such as domain adaptation (Bousmalis et al., 2016), style transfer (Karras et al., 2019b), inpainting (Pathak et al., 2016) and talking head generation (Zakharov et al., 2019).
Revised: GANs are used in many tasks such as domain adaptation (Bousmalis et al., 2016), style transfer (Karras et al., 2019b), inpainting (Pathak et al., 2016) and talking head generation (Zakharov et al., 2019).

Original: Unpaired image-to-image translation considers the tasks of transforming an image from a domain
Revised: Unpaired image-to-image translation considers the tasks of transforming an image from a domain

Original: A into an image in a domain B .
Revised: A into an image in a domain B .

Original: The training set comprises a sample of images from domains A and
Revised: The training set comprises a sample of images from domains A and

Original: B , but no pairs of corresponding images.
Revised: B , but no pairs of corresponding images.

Original: A classical approach is to train two generators ( A !
Revised: A classical approach is to train two generators ( A !

Original: B and
Revised: B and

Original: B !
Revised: B !

Original: A ) and two discriminators, one for each domain.
Revised: A ) and two discriminators, one for each domain.

Original: When there is a shared latent space between the domains, a possible choice is to use a variational auto encoder like in CoGAN Liu & Tuzel (2016).
Revised: When there is a shared latent space between the domains, a possible choice is to use a variational auto encoder like in CoGAN Liu & Tuzel (2016).

Original: CycleGAN Zhu et al.
Revised: CycleGAN Zhu et al.

Original: (2017), DualGAN Yi et al.
Revised: (2017), DualGAN Yi et al.

Original: (2017) and subsequent works (Liu et al., 2019;
Revised: (2017) and subsequent works (Liu et al., 2019;

Original: Fu et al., 2019; Choi et al., 2020) augment the adversarial loss induced by the discriminators with a cycle consistency constraint to preserve semantic information throughout the domain changes.
Revised: Fu et al., 2019; Choi et al., 2020) augment the adversarial loss induced by the discriminators with a cycle consistency constraint to preserve semantic information throughout the domain changes.

Original: All these variants have architectures roughly similar to CycleGAN: an encoder, a decoder and residual blocks operating on the latent space.
Revised: All these variants have architectures roughly similar to CycleGAN: an encoder, a decoder and residual blocks operating on the latent space.

Original: They also incorporate elements of other networks such as
Revised: They also incorporate elements of other networks such as

Original: StyleGAN Karras et al. (2019a).
Revised: StyleGAN Karras et al. (2019a).

Original: In our work, we build upon a simpliﬁed form of the CycleGAN architecture that generalizes over tasks easily.
Revised: In our work, we build upon a simpliﬁed form of the CycleGAN architecture that generalizes over tasks easily.

Original: We applied also our method to NICE-GAN Chen et al.
Revised: We applied also our method to NICE-GAN Chen et al.

Original: A concurrent work, GANHopper by Lira et al.
Revised: A concurrent work, GANHopper by Lira et al.

Original: (2020) proposes to iterate CycleGAN generators in order to perform the transformation.
Revised: (2020) proposes to iterate CycleGAN generators in order to perform the transformation.

Original: However, their method has many difference with ours: They iterate full generators and not a single residual block, their encoder and decoder are not ﬁxed, their number of iteration is ﬁxed and they have to use additional discriminators to act on intermediate transformation states.
Revised: However, their method has many difference with ours: They iterate full generators and not a single residual block, their encoder and decoder are not ﬁxed, their number of iteration is ﬁxed and they have to use additional discriminators to act on intermediate transformation states.

Original: Other works (Zhang et al., 2019; Li et al., 2020) use recurrent networks to perform transformations but this is done in a paired context, which results in very different methods from ours and GANHopper.
Revised: Other works (Zhang et al., 2019; Li et al., 2020) use recurrent networks to perform transformations but this is done in a paired context, which results in very different methods from ours and GANHopper.

Original: Transformation modulation is an interpolation between two image domains.
Revised: Transformation modulation is an interpolation between two image domains.

Original: It is a byproduct of some approaches:For instance, a linear interpolation in latent space (Brock et al., 2018; Radford et al., 2015)
Revised: It is a byproduct of some approaches:For instance, a linear interpolation in latent space (Brock et al., 2018; Radford et al., 2015)

Original: morphs between two images.
Revised: morphs between two images.

Original: Nevertheless, one important limitation is that the starting and ending points must both be known, which is not the case in unpaired learning.
Revised: Nevertheless, one important limitation is that the starting and ending points must both be known, which is not the case in unpaired learning.

Original: Other approaches such as the Fader networks (Lample et al., 2017) or StyleGan2 (Viazovetskyi et al., 2020) act on scalar or boolean attributes that are disentangled in the latent space ( eg., age for face images, wear glasses or not, etc).
Revised: Other approaches such as the Fader networks (Lample et al., 2017) or StyleGan2 (Viazovetskyi et al., 2020) act on scalar or boolean attributes that are disentangled in the latent space ( eg., age for face images, wear glasses or not, etc).

Original: Nevertheless, this results in complex models, for which dataset size and the variability of images strongly impacts the performance: they fail to modulate the transform with small datasets or with large variabilities.
Revised: Nevertheless, this results in complex models, for which dataset size and the variability of images strongly impacts the performance: they fail to modulate the transform with small datasets or with large variabilities.

Original: A comparison of PoL with the Fader network is provided in Appendix C and shows that our approach is more effective.
Revised: A comparison of PoL with the Fader network is provided in Appendix C and shows that our approach is more effective.

Original: Progressive learning and inference time modulation are performed in multi-scale methods such as SinGAN (Rott Shaham et al., 2019) and ProgressiveGAN (Karras et al., 2017).
Revised: Progressive learning and inference time modulation are performed in multi-scale methods such as SinGAN (Rott Shaham et al., 2019) and ProgressiveGAN (Karras et al., 2017).

Original: Progressive learning obtains excellent results for high resolution images where it is more difﬁcult to use classical approaches.
Revised: Progressive learning obtains excellent results for high resolution images where it is more difﬁcult to use classical approaches.

Original: The training is performed in several steps during which the size of both the images and the network are increased.
Revised: The training is performed in several steps during which the size of both the images and the network are increased.

Original: The inference time of some architectures can be modulated by stopping the forward pass at some layer (Huang et al., 2016; Wu et al., 2017; Veit & Belongie, 2017).This differs from our approach, where the number of residual block compositions (“powers”) can be chosen, to shorten the inference.
Revised: The inference time of some architectures can be modulated by stopping the forward pass at some layer (Huang et al., 2016; Wu et al., 2017; Veit & Belongie, 2017).This differs from our approach, where the number of residual block compositions (“powers”) can be chosen, to shorten the inference.

Original: A by-product is a reduction of the number of network parameters.
Revised: A by-product is a reduction of the number of network parameters.

Original: Weight sharing is a way of looking at our method, because the same layer is applied several times within the same network.
Revised: Weight sharing is a way of looking at our method, because the same layer is applied several times within the same network.

Original: Recurrent Neural Networks (RNN) are the most classical example weight sharing in a recursive architecture.
Revised: Recurrent Neural Networks (RNN) are the most classical example weight sharing in a recursive architecture.

Original: Besides RNNs, weight sharing is mainly used for model compression, sequential data and ordinary differential equations (ODE) (Gao et al., 2019; Han et al., 2015;
Revised: Besides RNNs, weight sharing is mainly used for model compression, sequential data and ordinary differential equations (ODE) (Gao et al., 2019; Han et al., 2015;

Original: Chen et al.
Revised: Chen et al.

Original: , 2018).A few works (Jastrzebski et al., 2017; Zhang & Jung, 2018) apply weight sharing to unfold a ResNet and evaluate its performance in classiﬁcation tasks.
Revised: , 2018).A few works (Jastrzebski et al., 2017; Zhang & Jung, 2018) apply weight sharing to unfold a ResNet and evaluate its performance in classiﬁcation tasks.

Original: The optimization is inherently difﬁcult, so they use independent batch normalization for each shared layer.
Revised: The optimization is inherently difﬁcult, so they use independent batch normalization for each shared layer.

Original: With PoL we observe the same optimization issues, that we solve by a progressive training strategy, see Section 3.3.
Revised: With PoL we observe the same optimization issues, that we solve by a progressive training strategy, see Section 3.3.

Original: Recent work (Jeon et al., 2020) are interested in the composition of the same block by considering the parallel with the ﬁxed point theorem, nevertheless their application remains to rather simple problems compared to our unpaired image-to-image translation tasks.
Revised: Recent work (Jeon et al., 2020) are interested in the composition of the same block by considering the parallel with the ﬁxed point theorem, nevertheless their application remains to rather simple problems compared to our unpaired image-to-image translation tasks.

Original:  P OWER OF LAYERS
Revised:  P OWER OF LAYERS

Original: We adopt the same context as CycleGANand focus on unpaired image-to-image translation: the objective is to transform an image from domain A into an image from domain B .
Revised: We adopt the same context as CycleGANand focus on unpaired image-to-image translation: the objective is to transform an image from domain A into an image from domain B .

Original: In our case the domains can be noise levels, painting styles, blur, JPEG artifacts, or simply object classes that appear in the image.
Revised: In our case the domains can be noise levels, painting styles, blur, JPEG artifacts, or simply object classes that appear in the image.

Original: The training is unpaired: we do not have pairs of corresponding images at training time.
Revised: The training is unpaired: we do not have pairs of corresponding images at training time.

Original: CycleGAN is simple, adaptable to different tasks and allows a direct comparison in Sections 4 and 5.
Revised: CycleGAN is simple, adaptable to different tasks and allows a direct comparison in Sections 4 and 5.

Original: We learn two generators and two discriminators.
Revised: We learn two generators and two discriminators.

Original: The generator G AB : I !
Revised: The generator G AB : I !

Original: I transforms an element of A into an element of B , and G BA goes the other way round, I being the ﬁxed-resolution image space.
Revised: I transforms an element of A into an element of B , and G BA goes the other way round, I being the ﬁxed-resolution image space.

Original: The discriminators D A : I !
Revised: The discriminators D A : I !

Original: [0 , 1] (resp.
Revised: [0 , 1] (resp.

Original: D B ) predicts whether an element belongs to domain
Revised: D B ) predicts whether an element belongs to domain

Original: A (resp B ).
Revised: A (resp B ).

Original: We use the same losses as commonly used in unpaired image-to-image translation:
Revised: We use the same losses as commonly used in unpaired image-to-image translation:

Original: [Equation]
Revised: [Equation]

Original: where L Adv ( G AB , D B ) =
Revised: where L Adv ( G AB , D B ) =

Original: E b ⇠ B [log D B ( b )]
Revised: E b ⇠ B [log D B ( b )]

Original: + E a ⇠ A [log(1  D B ( G AB ( a )))] ,
Revised: + E a ⇠ A [log(1  D B ( G AB ( a )))] ,

Original: L Cyc ( G AB , G BA ) =
Revised: L Cyc ( G AB , G BA ) =

Original: E b ⇠ B [ k G AB ( G BA ( b ))
Revised: E b ⇠ B [ k G AB ( G BA ( b ))

Original:  b k 1 ] + E a ⇠ A [ k G BA ( G AB ( a ))  a k 1 ] ,
Revised:  b k 1 ] + E a ⇠ A [ k G BA ( G AB ( a ))  a k 1 ] ,

Original: L Id ( G AB , G BA ) =
Revised: L Id ( G AB , G BA ) =

Original: E b ⇠ B [ k G AB ( b )  b k 2 ] + E a ⇠ A [ k G BA ( a )  a k 2 ] .
Revised: E b ⇠ B [ k G AB ( b )  b k 2 ] + E a ⇠ A [ k G BA ( a )  a k 2 ] .

Original: The Adversarial loss L Adv ( G AB , D B ) veriﬁes that the generated images are in the correct domain.
Revised: The Adversarial loss L Adv ( G AB , D B ) veriﬁes that the generated images are in the correct domain.

Original: The Cycle Consistency loss L Cyc ( G AB , G BA ) ensures a round-trip through the two generators reconstructs the initial image, and the identity loss L Id ( G AB , G BA ) penalizes the generators transforming images that are already in their target domain.
Revised: The Cycle Consistency loss L Cyc ( G AB , G BA ) ensures a round-trip through the two generators reconstructs the initial image, and the identity loss L Id ( G AB , G BA ) penalizes the generators transforming images that are already in their target domain.

Original: We keep the same linear combination coefﬁcients as in
Revised: We keep the same linear combination coefﬁcients as in

Original: CycleGAN:  Adv = 1 ,  Cyc = 10 ,  Id = 5 .
Revised: CycleGAN:  Adv = 1 ,  Cyc = 10 ,  Id = 5 .

Original: 3. N ETWORK ARCHITECTURE
Revised: 3. N ETWORK ARCHITECTURE

Original: We start from the CycleGAN architecture.
Revised: We start from the CycleGAN architecture.

Original: The encoder and decoder consist of 2 layers and a residual block.
Revised: The encoder and decoder consist of 2 layers and a residual block.

Original: The embedding space E of our model is 256 ⇥ 64 ⇥ 64 : its spatial resolution is 1/4 the input image resolution of 256 ⇥ 256 and it has 256 channels.
Revised: The embedding space E of our model is 256 ⇥ 64 ⇥ 64 : its spatial resolution is 1/4 the input image resolution of 256 ⇥ 256 and it has 256 channels.

Original: All translation operations take place in the ﬁxed embedding space E .
Revised: All translation operations take place in the ﬁxed embedding space E .

Original: The encoder Enc : I !
Revised: The encoder Enc : I !

Original: E produces the embedding and consists of two convolutions.
Revised: E produces the embedding and consists of two convolutions.

Original: The decoder
Revised: The decoder

Original: Dec :
Revised: Dec :

Original: E !
Revised: E !

Original: I turns the embedding back to image space and consists of two transposed convolutions.
Revised: I turns the embedding back to image space and consists of two transposed convolutions.

Original: Pre-training of the auto-encoder.
Revised: Pre-training of the auto-encoder.

Original: We train the encoder and decoder of our model on a reconstruction task using an ` 2 reconstruction loss in pixel space.
Revised: We train the encoder and decoder of our model on a reconstruction task using an ` 2 reconstruction loss in pixel space.

Original: We use the Adam optimizer.
Revised: We use the Adam optimizer.

Original: Our data-augmentation consists of an image resizing, a random crop and a random horizontal ﬂip.
Revised: Our data-augmentation consists of an image resizing, a random crop and a random horizontal ﬂip.

Original: Both the encoder and decoder weights are ﬁxed for all the other tasks, only the residual block is adapted (and the discriminator in case we use it for the stopping criterion).
Revised: Both the encoder and decoder weights are ﬁxed for all the other tasks, only the residual block is adapted (and the discriminator in case we use it for the stopping criterion).

Original: There are two choices to train the auto-encoder:
Revised: There are two choices to train the auto-encoder:

Original: (1) train with 6M unlabeled images randomly drawn from the YFCC100M dataset (Thomee et al., 2016) during one epoch, which makes it independent of the translation training dataset; (2) train on the dataset of our unpaired image-to-image translation task.
Revised: (1) train with 6M unlabeled images randomly drawn from the YFCC100M dataset (Thomee et al., 2016) during one epoch, which makes it independent of the translation training dataset; (2) train on the dataset of our unpaired image-to-image translation task.

Original: For simplicity, we choose the ﬁrst option and use a single encoder and decoder for all the tasks presented in this article.
Revised: For simplicity, we choose the ﬁrst option and use a single encoder and decoder for all the tasks presented in this article.

Original: For the NiceGAN experiments, the encoder and decoder are trained jointly with the discriminator, so the auxiliary dataset YFCC100M is not used.
Revised: For the NiceGAN experiments, the encoder and decoder are trained jointly with the discriminator, so the auxiliary dataset YFCC100M is not used.

Original: The embedding transformer – single block.
Revised: The embedding transformer – single block.

Original: The transformation between domains is based on a residual block f AB , similar to the feed-forward network used in transformers (Vaswani et al., 2017).
Revised: The transformation between domains is based on a residual block f AB , similar to the feed-forward network used in transformers (Vaswani et al., 2017).

Original: It writes:
Revised: It writes:

Original: [Equation]
Revised: [Equation]

Original: There is a dimensionality expansion factor K between the two convolutions in the residual block (see Figure 1).
Revised: There is a dimensionality expansion factor K between the two convolutions in the residual block (see Figure 1).

Original: Adjusting K changes the model’s capacity.
Revised: Adjusting K changes the model’s capacity.

Original: We adopt the now standard choice of the original transformer paper ( K = 4 ).
Revised: We adopt the now standard choice of the original transformer paper ( K = 4 ).

Original: The full generator writes
Revised: The full generator writes

Original: [Equation]
Revised: [Equation]

Original: The other direction, with f BA and res BA , is deﬁned accordingly.
Revised: The other direction, with f BA and res BA , is deﬁned accordingly.

Original: Powers of layers.
Revised: Powers of layers.

Original: We start from the architecture above and augment its representation capacity.
Revised: We start from the architecture above and augment its representation capacity.

Original: There are two standard ways of doing this: (1) augmenting the capacity of the f AB block by increasing
Revised: There are two standard ways of doing this: (1) augmenting the capacity of the f AB block by increasing

Original: K ; (2) increasing the depth of the network by chaining several instances of f AB , since the intermediate representations are compatible.
Revised: K ; (2) increasing the depth of the network by chaining several instances of f AB , since the intermediate representations are compatible.

Original: In contrast to these ﬁxed architectures, PoL iterates the f AB block n  1 times, which amounts to sharing the weights of a deeper network:
Revised: In contrast to these ﬁxed architectures, PoL iterates the f AB block n  1 times, which amounts to sharing the weights of a deeper network:

Original: [Equation]
Revised: [Equation]

Original: 3. O PTIMIZATION IN A RESIDUALS BLOCKS WEIGHT SHARING CONTEXT
Revised: 3. O PTIMIZATION IN A RESIDUALS BLOCKS WEIGHT SHARING CONTEXT

Original: In the following, we drop the AB sufﬁx from f AB , since powers of layers operates in the same way on f AB and f BA .
Revised: In the following, we drop the AB sufﬁx from f AB , since powers of layers operates in the same way on f AB and f BA .

Original: Thus, f : E !
Revised: Thus, f : E !

Original: E is f ( x ) = x
Revised: E is f ( x ) = x

Original: + res( x ) .
Revised: + res( x ) .

Original: The parameters of f are collected in a vector w .
Revised: The parameters of f are collected in a vector w .

Original: The embedding x 2 E is a 3D activation map, but for the sake of the mathematical derivation we linearize it to a vector.
Revised: The embedding x 2 E is a 3D activation map, but for the sake of the mathematical derivation we linearize it to a vector.

Original: The partial derivatives of f are @ f @ x = @ res @
Revised: The partial derivatives of f are @ f @ x = @ res @

Original: x
Revised: x

Original: + Id and @ f @ w = @ res @ w .
Revised: + Id and @ f @ w = @ res @ w .

Original: We compose the f function n times as
Revised: We compose the f function n times as

Original: [Equation]
Revised: [Equation]

Original: The stability of the SGD optimization depends on the magnitude and conditioning of the matrix:
Revised: The stability of the SGD optimization depends on the magnitude and conditioning of the matrix:

Original: [Equation]
Revised: [Equation]

Original: which is sensitive to initialization during the ﬁrst optimization epochs.
Revised: which is sensitive to initialization during the ﬁrst optimization epochs.

Original: Indeed, the length of the SGD steps on w depends on the eigenvalues of M n .
Revised: Indeed, the length of the SGD steps on w depends on the eigenvalues of M n .

Original: When simplifying the basic residual block to a linear transformation L 2 R d ⇥ d (i.e., ignoring the normalization and the ReLU non-linearity), we have
Revised: When simplifying the basic residual block to a linear transformation L 2 R d ⇥ d (i.e., ignoring the normalization and the ReLU non-linearity), we have

Original: M n =
Revised: M n =

Original: ( L + Id) n  1 .
Revised: ( L + Id) n  1 .

Original: The eigenvalues of M n are (  i + 1) n  1 , where  1 , ...,  d are the eigenvalues of
Revised: The eigenvalues of M n are (  i + 1) n  1 , where  1 , ...,  d are the eigenvalues of

Original: L .
Revised: L .

Original: At initialization, the components of L are sampled from a random uniform distribution.
Revised: At initialization, the components of L are sampled from a random uniform distribution.

Original: To reduce the magnitude of  i , one option is to make the entries of L small.
Revised: To reduce the magnitude of  i , one option is to make the entries of L small.

Original: However, to decrease (  i + 1) n sufﬁciently,  i must be so small that it introduces ﬂoating-point cancellations when the residual block is added back to the shortcut connection.
Revised: However, to decrease (  i + 1) n sufﬁciently,  i must be so small that it introduces ﬂoating-point cancellations when the residual block is added back to the shortcut connection.

Original: This is why we prefer to adjust n , as detailed next.
Revised: This is why we prefer to adjust n , as detailed next.

Original: 3. P ROGRESSIVE TRAINING
Revised: 3. P ROGRESSIVE TRAINING

Original: We adopt a progressive learning schedule in a “warm up” phase: we start the optimization with a single block and add one iteration at every epoch until we reach the required n blocks.
Revised: We adopt a progressive learning schedule in a “warm up” phase: we start the optimization with a single block and add one iteration at every epoch until we reach the required n blocks.

Original: This is
Revised: This is

Original: [Table]
Revised: [Table]

Original: Appendix 
Revised: Appendix A reports standard deviations and more results.

Original: A reports standard deviations and more results.

Original: possible because the blocks operate in the same embedding space E and because their weights are shared
Revised: possible because the blocks operate in the same embedding space E and because their weights are shared

Original: , so all blocks are still in the same learning schedule.
Revised: , so all blocks are still in the same learning schedule.

Original: This approach avoids the numerical exponentiation effects at initialization mentioned in the previous paragraph.
Revised: This approach avoids the numerical exponentiation effects at initialization mentioned in the previous paragraph.

Original: After the warm-up phase, the gradient descent starts to converge, so these numerical effects become unlikely.
Revised: After the warm-up phase, the gradient descent starts to converge, so these numerical effects become unlikely.

Original: In addition, this approach allows the discriminator to improve progressively during the training.
Revised: In addition, this approach allows the discriminator to improve progressively during the training.

Original: For example, in the case of the transformation horse !
Revised: For example, in the case of the transformation horse !

Original: zebra, a slightly whitened horse fools the discriminator at the beginning of the training, but a texture with stronger stripes is required later on.
Revised: zebra, a slightly whitened horse fools the discriminator at the beginning of the training, but a texture with stronger stripes is required later on.

Original: Training for modulation.
Revised: Training for modulation.

Original: If the network is trained with a ﬁxed number of compositions, the intermediate states do not correspond to modulations of the transformation that “look right” (see
Revised: If the network is trained with a ﬁxed number of compositions, the intermediate states do not correspond to modulations of the transformation that “look right” (see

Original: Appendix A).
Revised: Appendix A).

Original: Therefore, in addition to this scheduled number of iterations during the ﬁrst n epochs of warm-up, we also randomize the number of iterations in subsequent epochs.
Revised: Therefore, in addition to this scheduled number of iterations during the ﬁrst n epochs of warm-up, we also randomize the number of iterations in subsequent epochs.

Original: This forces the generator to also produce acceptable intermediate states, and enables modulating the transform.
Revised: This forces the generator to also produce acceptable intermediate states, and enables modulating the transform.

Original: Stopping criterion at inference time.
Revised: Stopping criterion at inference time.

Original: Each image of domain A is more or less close to domain B .
Revised: Each image of domain A is more or less close to domain B .

Original: For example, when denoising, the noise level can vary so the denoising strength should adapt to the input.
Revised: For example, when denoising, the noise level can vary so the denoising strength should adapt to the input.

Original: Similarly for horse !
Revised: Similarly for horse !

Original: zebra: a white horse is closer to a zebra than a brown horse.
Revised: zebra: a white horse is closer to a zebra than a brown horse.

Original: Therefore, at inference time, we can adjust the number of compositions as well.
Revised: Therefore, at inference time, we can adjust the number of compositions as well.

Original: In particular, for each test image, we select n that best deceives the discriminator, thus effectively adapting the processing to the relative distance to the target domain.
Revised: In particular, for each test image, we select n that best deceives the discriminator, thus effectively adapting the processing to the relative distance to the target domain.

Original:  A NALYSIS
Revised:  A NALYSIS

Original: In this section we study the impact of the main training and design choices and on the performance of powers of layers.
Revised: In this section we study the impact of the main training and design choices and on the performance of powers of layers.

Original: Appendices A and B provide complementary analysis for training- and inferencetime choices, respectively.
Revised: Appendices A and B provide complementary analysis for training- and inferencetime choices, respectively.

Original: For this preliminary analysis, we focus on denoising tasks for which the performance is easily measurable.
Revised: For this preliminary analysis, we focus on denoising tasks for which the performance is easily measurable.

Original: We add three types of noise to images: Gaussian noise, Gaussian blur and JPEG artifacts.
Revised: We add three types of noise to images: Gaussian noise, Gaussian blur and JPEG artifacts.

Original: The noise intensity is quantiﬁed by the noise standard deviation, the blur radius and the JPEG quality factor, respectively.
Revised: The noise intensity is quantiﬁed by the noise standard deviation, the blur radius and the JPEG quality factor, respectively.

Original: We generate transformed images and measure how well our method recovers the initial image.
Revised: We generate transformed images and measure how well our method recovers the initial image.

Original: This provides a more reliable accuracy measure than a purely unpaired evaluation, that will be carried out in the experimental section.
Revised: This provides a more reliable accuracy measure than a purely unpaired evaluation, that will be carried out in the experimental section.

Original: Note that, in the literature, these tasks are best addressed by providing (original, noisy) pairs if images.
Revised: Note that, in the literature, these tasks are best addressed by providing (original, noisy) pairs if images.

Original: Our objective is to remain in a completely unpaired setting during the training phase.
Revised: Our objective is to remain in a completely unpaired setting during the training phase.

Original: It corresponds to the case where parallel data is not available (like for the restoration of ancient movies), and also better reﬂects the situation where the noise strength is not known in advance.
Revised: It corresponds to the case where parallel data is not available (like for the restoration of ancient movies), and also better reﬂects the situation where the noise strength is not known in advance.

Original: Therefore, the original image is solely employed to measure the performance.
Revised: Therefore, the original image is solely employed to measure the performance.

Original: Experimental protocol.
Revised: Experimental protocol.

Original: To train, we sample 800 domain A images from the high-resolution Div2K dataset (Agustsson & Timofte, 2017).
Revised: To train, we sample 800 domain A images from the high-resolution Div2K dataset (Agustsson & Timofte, 2017).

Original: In the baseline training procedure, the warm-up phase starts from a single block and increases the number of compositions at every epoch, until we reach epoch n tr .
Revised: In the baseline training procedure, the warm-up phase starts from a single block and increases the number of compositions at every epoch, until we reach epoch n tr .

Original: Then we keep the number of compositions ﬁxed.
Revised: Then we keep the number of compositions ﬁxed.

Original: We test on the Urban-100 (Huang et al., 2015) dataset.
Revised: We test on the Urban-100 (Huang et al., 2015) dataset.

Original: Unless speciﬁed otherwise, we set the number of compositions to n te = n tr and measure the Peak Signal to Noise Ratio (PSNR) of our model on the dataset images, degraded with the same intensity as at training time.
Revised: Unless speciﬁed otherwise, we set the number of compositions to n te = n tr and measure the Peak Signal to Noise Ratio (PSNR) of our model on the dataset images, degraded with the same intensity as at training time.

Original: For the JPEG case we use the Naturalness Image Quality Evaluator metric (NIQE, lower=better) instead, because it is more sensitive to JPEG artifacts (Mittal et al., 2013).
Revised: For the JPEG case we use the Naturalness Image Quality Evaluator metric (NIQE, lower=better) instead, because it is more sensitive to JPEG artifacts (Mittal et al., 2013).

Original: NIQE is a perceptual metric that does not take the original image into account (i.e. for JPEG we emphdo use an unpaired metric).
Revised: NIQE is a perceptual metric that does not take the original image into account (i.e. for JPEG we emphdo use an unpaired metric).

Original: Block composition or independent successive blocks?
Revised: Block composition or independent successive blocks?

Original: Table 1 compares PoL’s composition of the same block versus using independent blocks with distinct weights.
Revised: Table 1 compares PoL’s composition of the same block versus using independent blocks with distinct weights.

Original: In spite of the much larger
Revised: In spite of the much larger

Original: [Figure]
Revised: [Figure]

Original: F i x e d n tr =
Revised: F i x e d n tr =

Original: [Figure]
Revised: [Figure]

Original: R a ndo m n tr 2 [[ 20 , 30 ]] capacity by independent blocks reduction
Revised: R a ndo m n tr 2 [[ 20 , 30 ]] capacity by independent blocks reduction

Original: Our interpretation is that the model is easier to train.
Revised: Our interpretation is that the model is easier to train.

Original: Analysis of the progressive training strategy.
Revised: Analysis of the progressive training strategy.

Original: Table 1 also evaluates the impact of the maximum number of compositions n tr .
Revised: Table 1 also evaluates the impact of the maximum number of compositions n tr .

Original: Having several compositions clearly helps.
Revised: Having several compositions clearly helps.

Original: Since we choose the number of compositions n te at inference time (see next paragraph), it may be relevant to vary n tr at training time to minimize the discrepancy between the train- and test-time settings.
Revised: Since we choose the number of compositions n te at inference time (see next paragraph), it may be relevant to vary n tr at training time to minimize the discrepancy between the train- and test-time settings.

Original: For this, we tried different intervals to randomly draw the maximum number of compositions for each epoch, after the warm-up phase.
Revised: For this, we tried different intervals to randomly draw the maximum number of compositions for each epoch, after the warm-up phase.

Original: If n te is ﬁxed, the optimal choice is n tr = n te .
Revised: If n te is ﬁxed, the optimal choice is n tr = n te .

Original: However, if we use an adaptive n te , the best range is n tr 2 [[20 , 30]] , and the adaptive case with randomised training gives the best performance for denoising and debluring.
Revised: However, if we use an adaptive n te , the best range is n tr 2 [[20 , 30]] , and the adaptive case with randomised training gives the best performance for denoising and debluring.

Original: Appendix A reports results obtained with different n tr ranges.
Revised: Appendix A reports results obtained with different n tr ranges.

Original: Stopping criterion.
Revised: Stopping criterion.

Original: We consider two cases: either we use a ﬁxed n te , or we use the discriminator to evaluate the transformation quality: it selects the value n te maximizing the target discriminator error for a given image.
Revised: We consider two cases: either we use a ﬁxed n te , or we use the discriminator to evaluate the transformation quality: it selects the value n te maximizing the target discriminator error for a given image.

Original: Figure 2 shows that setting a ﬁxed n tr causes the discriminator to select n te = n tr as the best iteration at inference time.
Revised: Figure 2 shows that setting a ﬁxed n tr causes the discriminator to select n te = n tr as the best iteration at inference time.

Original: By selecting the best n te for each image we obtain on average a PSNR improvement of +1.36dB for a Gaussian noise of standard deviation 30 , compared to ﬁxing n te .
Revised: By selecting the best n te for each image we obtain on average a PSNR improvement of +1.36dB for a Gaussian noise of standard deviation 30 , compared to ﬁxing n te .

Original: In Appendix B, we compare it with the best possible stopping criterion: an oracle that selects n te directly on the PSNR.
Revised: In Appendix B, we compare it with the best possible stopping criterion: an oracle that selects n te directly on the PSNR.

Original: Our adaptive strategy signiﬁcantly tightens the gap to this upper bound.
Revised: Our adaptive strategy signiﬁcantly tightens the gap to this upper bound.

Original: Comparison with CycleGAN.
Revised: Comparison with CycleGAN.

Original: We use CycleGAN as a baseline.
Revised: We use CycleGAN as a baseline.

Original: The differences between CycleGAN and powers of layers are (1) we use a single encoder and decoder trained in advance and common to all tasks; (2) CycleGAN has 9 residual blocks, PoL iterates a single residual block an arbitrary number of times.
Revised: The differences between CycleGAN and powers of layers are (1) we use a single encoder and decoder trained in advance and common to all tasks; (2) CycleGAN has 9 residual blocks, PoL iterates a single residual block an arbitrary number of times.

Original: The inference time of PoL depends on the number of compositions n te but the number of parameters does not:
Revised: The inference time of PoL depends on the number of compositions n te but the number of parameters does not:

Original: [Table]
Revised: [Table]

Original: Figure 3 compares the performance obtained by the two methods on denoising tasks with varying noise intensities.
Revised: Figure 3 compares the performance obtained by the two methods on denoising tasks with varying noise intensities.

Original: PoL gives better results than CycleGAN in terms of objective metrics, and overall the images produced by our method look as realistic and/or accurate.
Revised: PoL gives better results than CycleGAN in terms of objective metrics, and overall the images produced by our method look as realistic and/or accurate.

Original: Training with few images.
Revised: Training with few images.

Original: Figure 3 also compares our method with CycleGAN when training in a data-starving scenario.
Revised: Figure 3 also compares our method with CycleGAN when training in a data-starving scenario.

Original: Whatever the number of training images, PoL outperforms CycleGAN, but the gap is particularly important with very few images.
Revised: Whatever the number of training images, PoL outperforms CycleGAN, but the gap is particularly important with very few images.

Original: This is expected for two reasons.
Revised: This is expected for two reasons.

Original: Firstly, our approach has fewer parameters to learn than CycleGAN.
Revised: Firstly, our approach has fewer parameters to learn than CycleGAN.

Original: Secondly, it only requires to learn the transformation because the encoder and decoder are pre-learned as a vanilla auto-encoder, while
Revised: Secondly, it only requires to learn the transformation because the encoder and decoder are pre-learned as a vanilla auto-encoder, while

Original: CycleGAN needs to learn how to encode and decode images.
Revised: CycleGAN needs to learn how to encode and decode images.

Original: Parametrization: remarks.
Revised: Parametrization: remarks.

Original: Beyond the settings inherited from CycleGAN, the main training parameters of Powers of layers are the maximum number of compositions n tr and the range from which they are randomly sampled.
Revised: Beyond the settings inherited from CycleGAN, the main training parameters of Powers of layers are the maximum number of compositions n tr and the range from which they are randomly sampled.

Original: The number of compositions at inference time n te is also important but the discriminator criterion can be used to set it automatically.
Revised: The number of compositions at inference time n te is also important but the discriminator criterion can be used to set it automatically.

Original: [Table]
Revised: [Table]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: We now run experiments on two image generation applications.
Revised: We now run experiments on two image generation applications.

Original: We refer to Section 3.1 for the architecture and training protocol.
Revised: We refer to Section 3.1 for the architecture and training protocol.

Original: In Appendix C we also give a comparison with the Fader network for the capacity to modulate a transformation, and more visual examples in Appendix D.
Revised: In Appendix C we also give a comparison with the Fader network for the capacity to modulate a transformation, and more visual examples in Appendix D.

Original: Unpaired image-to-image translation.
Revised: Unpaired image-to-image translation.

Original: We report results for 6 of the 8 unpaired image-to-image translation tasks introduced in the CycleGAN (Zhu et al., 2017) paper (the two remaining ones lead to the same conclusions) and we used the datasets from their website.
Revised: We report results for 6 of the 8 unpaired image-to-image translation tasks introduced in the CycleGAN (Zhu et al., 2017) paper (the two remaining ones lead to the same conclusions) and we used the datasets from their website.

Original: We compare the Frechet Inception
Revised: We compare the Frechet Inception

Original: Distance (FID) of CycleGAN, NiceGAN and our approach applied to this method in Figure 4.
Revised: Distance (FID) of CycleGAN, NiceGAN and our approach applied to this method in Figure 4.

Original: The
Revised: The

Original: FID measures the similarity between two datasets of images, we use it to compare the target dataset with the transformed dataset.
Revised: FID measures the similarity between two datasets of images, we use it to compare the target dataset with the transformed dataset.

Original: It is a noisy measure for which only large deviations are signiﬁcant.
Revised: It is a noisy measure for which only large deviations are signiﬁcant.

Original: Yet the results and visualization show that our method has results comparable to those of CycleGAN and
Revised: Yet the results and visualization show that our method has results comparable to those of CycleGAN and

Original: NiceGAN, achieved with much fewer parameters.
Revised: NiceGAN, achieved with much fewer parameters.

Original: High resolution experiments.
Revised: High resolution experiments.

Original: PoL is fully convolutional architecture, therefore it is technically possible to apply models trained in a low resolution to high resolution images.
Revised: PoL is fully convolutional architecture, therefore it is technically possible to apply models trained in a low resolution to high resolution images.

Original: However, the results
Revised: However, the results

Original: [Figure]
Revised: [Figure]

Original: are not always convincing, as shown in Figure 5 (right) where the model trained on low resolution images does not create stripes at the “right” scale on zebras.
Revised: are not always convincing, as shown in Figure 5 ( ) where the model trained on low resolution images does not create stripes at the “right” scale on zebras.

Original: To circumvent this problem, CycleGAN trains on patches taken from high resolution images.
Revised: To circumvent this problem, CycleGAN trains on patches taken from high resolution images.

Original: This works for transformations affecting the whole image (painting $ photo), but this is not applicable in the case where only a part of the image is affected (horse !
Revised: This works for transformations affecting the whole image (painting $ photo), but this is not applicable in the case where only a part of the image is affected (horse !

Original: zebra).
Revised: zebra).

Original: In contrast, our proposal can adapt the memory used by changing its number of compositions, so we can apply it to very large images without running out of memory.
Revised: In contrast, our proposal can adapt the memory used by changing its number of compositions, so we can apply it to very large images without running out of memory.

Original: Figure(left) depicts results obtained with our method trained on high resolution images.
Revised: Figure(left) depicts results obtained with our method trained on high resolution images.

Original: Combining transformation. 
Revised: Adjusting n te at inference time. Each image is more or less distant from the target domain, so we explore adapting the transformation to each image rather than applying a ﬁxed transformation. For example, depending on the amount of noise, we may want to adjust the strength of the denoising. By modulating n te we can adapt the transformation to each image.

Revised: Figure 6 shows that the more noisy the input image is, the more we should compose to best denoise with Powers-of-Layers.

Revised: In the same way, Figure 7 shows the progressive transformations obtained on different unpaired image-to-image translation tasks.

Revised: It shows that progressive transformations are realistic for most tasks.

Original:  The different blocks associated with different transformations operate in the same embedding space for different tasks.
Revised: [Figure] [Figure] Combining transformation. The different blocks associated with different transformations operate in the same embedding space for different tasks.

Original: Hence we can compose transformations, each being realized by one residual block.
Revised: Hence we can compose transformations, each being realized by one residual block.

Original: We train Transform #1 in the usual way, then freeze its residual block.
Revised: We train Transform #1 in the usual way, then freeze its residual block.

Original: Transform #2 is trained on the output of #1.
Revised: Transform #2 is trained on the output of #1.

Original: Visual results are in Figure 6.
Revised: Visual results are in Figure 8.

Original: The composition in the embedding space gives better results than decoding/encoding to image space mid-way.
Revised: The composition in the embedding space gives better results than decoding/encoding to image space mid-way.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: Powers of layers iterates a residual block to learns a complex transformation with no direct supervision.
Revised: Powers of layers iterates a residual block to learns a complex transformation with no direct supervision.

Original: On various tasks, It gives similar performance to CycleGAN and NiceGAN with fewer parameters.
Revised: On various tasks, It gives similar performance to CycleGAN and NiceGAN with fewer parameters.

Original: The ﬂexibility offered by the common embedding space allows the modulation of the transformation strength, or to compose several transformations.
Revised: The ﬂexibility offered by the common embedding space allows the modulation of the transformation strength, or to compose several transformations.

Original: While in most examples the discriminator is only used for training, Powers of layers can also exploit it to adjust the transformation to the input image at inference time.
Revised: While in most examples the discriminator is only used for training, Powers of layers can also exploit it to adjust the transformation to the input image at inference time.

Original: [Figure]
Revised: [Figure]