Original: MVP: M ULTIVARIATE POLYNOMIALS FOR CONDITIONAL DATA GENERATION
Revised: MVP: M ULTIVARIATE POLYNOMIALS FOR CONDITIONAL DATA GENERATION

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Conditional Generative Adversarial Nets (cGANs) have been widely adopted for image generation.
Revised: Conditional Generative Adversarial Nets (cGANs) have been widely adopted for image generation.

Original: cGANs take i) a noise vector and ii) a conditional variable as input.
Revised: cGANs take i) a noise vector and ii) a conditional variable as input.

Original: The conditional variable can be discrete (e.g., a class label) or continuous (e.g., an input image) resulting into class-conditional (image) generation and imageto-image translation models, respectively.
Revised: The conditional variable can be discrete (e.g., a class label) or continuous (e.g., an input image) resulting into class-conditional (image) generation and imageto-image translation models, respectively.

Original: However, depending on whether the conditional variable is discrete or continuous, various cGANs employ substantially different deep architectures and loss functions for their training.
Revised: However, depending on whether the conditional variable is discrete or continuous, various cGANs employ substantially different deep architectures and loss functions for their training.

Original: In this paper, we propose a novel framework, called MVP , for conditional data generation.
Revised: In this paper, we propose a novel framework, called MVP , for conditional data generation.

Original: MVP resorts to multivariate polynomials of higher-order and treats in a uniﬁed way both discrete and continuous conditional variables.
Revised: MVP resorts to multivariate polynomials of higher-order and treats in a uniﬁed way both discrete and continuous conditional variables.

Original: MVP is highly expressive, capturing higher-order auto- and cross-correlations of input variables (noise vector and conditional variable).
Revised: MVP is highly expressive, capturing higher-order auto- and cross-correlations of input variables (noise vector and conditional variable).

Original: Tailored sharing schemes are designed between the polynomial’s parameter tensors, which result in simple recursive formulas.
Revised: Tailored sharing schemes are designed between the polynomial’s parameter tensors, which result in simple recursive formulas.

Original: MVP can synthesize realistic images in both class-conditional and image-to-image translation tasks even in the absence of activation functions between the layers.
Revised: MVP can synthesize realistic images in both class-conditional and image-to-image translation tasks even in the absence of activation functions between the layers.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Modelling high-dimensional distributions and generating samples from complex distributions are fundamental tasks in machine learning.
Revised: Modelling high-dimensional distributions and generating samples from complex distributions are fundamental tasks in machine learning.

Original: Generative adversarial networks (GANs) (Goodfellow et al., 2014) have demonstrated spectacular results in the two tasks using both unsupervised (Miyato et al., 2018) and supervised (Brock et al., 2019) learning.
Revised: Generative adversarial networks (GANs) (Goodfellow et al., 2014) have demonstrated spectacular results in the two tasks using both unsupervised (Miyato et al., 2018) and supervised (Brock et al., 2019) learning.

Original: In the unsupervised setting, (the generator of) a GAN accepts as input a noise vector z
Revised: In the unsupervised setting, (the generator of) a GAN accepts as input a noise vector z

Original: I and maps the noise vector to a high-dimensional output.
Revised: I and maps the noise vector to a high-dimensional output.

Original: The supervised models, called conditional Generative Adversarial Nets (cGANs) (Mirza & Osindero, 2014), accept both a noise vector
Revised: The supervised models, called conditional Generative Adversarial Nets (cGANs) (Mirza & Osindero, 2014), accept both a noise vector

Original: z
Revised: z

Original: I and an additional conditional variable z II that facilitates the generation.
Revised: I and an additional conditional variable z II that facilitates the generation.

Original: The conditional variable can be discrete (e.g., a class or an attribute label) or continuous (e.g., a low-resolution image).
Revised: The conditional variable can be discrete (e.g., a class or an attribute label) or continuous (e.g., a low-resolution image).

Original: The impressive results obtained with both discrete conditional input (Brock et al., 2019) and continuous conditional input (Park et al., 2019; Ledig et al., 2017) have led to a plethora of applications that range from text-to-image synthesis (Qiao et al., 2019) to deblurring (Yan & Wang, 2017) and medical analysis (You et al., 2019).
Revised: The impressive results obtained with both discrete conditional input (Brock et al., 2019) and continuous conditional input (Park et al., 2019; Ledig et al., 2017) have led to a plethora of applications that range from text-to-image synthesis (Qiao et al., 2019) to deblurring (Yan & Wang, 2017) and medical analysis (You et al., 2019).

Original: Despite the similarity in the formulation for discrete and continuous conditional input (i.e., learning the function G p
Revised: Despite the similarity in the formulation for discrete and continuous conditional input (i.e., learning the function G p

Original: z
Revised: z

Original: I , z II q ), the literature has focused on substantially different architectures and losses.
Revised: I , z II q ), the literature has focused on substantially different architectures and losses.

Original: Frequently, techniques are simultaneously developed, e.g., the self-attention in the class-conditional
Revised: Frequently, techniques are simultaneously developed, e.g., the self-attention in the class-conditional

Original: Self-Attention GAN (Zhang et al., 2019) and in the Attention-GAN (Chen et al., 2018) with continuous conditional input.
Revised: Self-Attention GAN (Zhang et al., 2019) and in the Attention-GAN (Chen et al., 2018) with continuous conditional input.

Original: This delays the progress since practitioners develop twice as many architectures and losses for every case.
Revised: This delays the progress since practitioners develop twice as many architectures and losses for every case.

Original: A couple of straightforward ideas can be employed to unify the behavior of the two conditional variable types.
Revised: A couple of straightforward ideas can be employed to unify the behavior of the two conditional variable types.

Original: One idea is to use an encoder network to obtain representations that are independent of the conditional variable.
Revised: One idea is to use an encoder network to obtain representations that are independent of the conditional variable.

Original: This has two drawbacks: i) the network ignores the noise and a deterministic one-variable mapping is learned (Isola et al., 2017), ii) such encoder has not been successful so far for discrete conditional input.
Revised: This has two drawbacks: i) the network ignores the noise and a deterministic one-variable mapping is learned (Isola et al., 2017), ii) such encoder has not been successful so far for discrete conditional input.

Original: An alternative idea is to directly concatenate the labels in the latent space instead of ﬁnding an embedding.
Revised: An alternative idea is to directly concatenate the labels in the latent space instead of ﬁnding an embedding.

Original: In AC-GAN (Odena et al., 2017) the class labels are concatenated with the noise; however, the model does not scale well beyond 10 classes.
Revised: In AC-GAN (Odena et al., 2017) the class labels are concatenated with the noise; however, the model does not scale well beyond 10 classes.

Original: We argue that concatenation of the input is only capturing additive correlation and not higher-order interactions between the inputs.
Revised: We argue that concatenation of the input is only capturing additive correlation and not higher-order interactions between the inputs.

Original: A detailed discussion is conducted on sec.
Revised: A detailed discussion is conducted on sec.

Original: D (in the Appendix).
Revised: D (in the Appendix).

Original: A polynomial expansion with respect to the input variables can capture such higher-order correlations.
Revised: A polynomial expansion with respect to the input variables can capture such higher-order correlations.

Original: Π -Net (Chrysos et al., 2020) casts the function approximation into a polynomial expansion of a single input variable.
Revised: Π -Net (Chrysos et al., 2020) casts the function approximation into a polynomial expansion of a single input variable.

Original: By concatenating the input variables, we can express the function approximation as a polynomial of the fused variable.
Revised: By concatenating the input variables, we can express the function approximation as a polynomial of the fused variable.

Original: However, the concatenation reduces the ﬂexibility of the model signiﬁcantly, e.g., it enforces the same order of expansion with respect to the different variables and it only allows the same parameter sharing scheme to all variables.
Revised: However, the concatenation reduces the ﬂexibility of the model signiﬁcantly, e.g., it enforces the same order of expansion with respect to the different variables and it only allows the same parameter sharing scheme to all variables.

Original: We introduce a multivariate framework, called MVP , for conditional data generation.
Revised: We introduce a multivariate framework, called MVP , for conditional data generation.

Original: MVP resorts to multivariate polynomials with two input variables, i.e., z I for the noise vector and z II for the conditional variable.
Revised: MVP resorts to multivariate polynomials with two input variables, i.e., z I for the noise vector and z II for the conditional variable.

Original: MVP captures higher-order auto- and cross-correlations between the variables.
Revised: MVP captures higher-order auto- and cross-correlations between the variables.

Original: By imposing a tailored structure in the higher-order interactions, we obtain an intuitive, recursive formulation for MVP.
Revised: By imposing a tailored structure in the higher-order interactions, we obtain an intuitive, recursive formulation for MVP.

Original: The formulation is ﬂexible and enables different constraints to be applied to each variable and its associated parameters.
Revised: The formulation is ﬂexible and enables different constraints to be applied to each variable and its associated parameters.

Original: The formulation can be trivially extended to M input variables.
Revised: The formulation can be trivially extended to M input variables.

Original: In summary, our contributions are the following:
Revised: In summary, our contributions are the following:

Original: • We introduce a framework, called MVP, that expresses a high-order, multivariate polynomial for conditional data generation.
Revised: • We introduce a framework, called MVP, that expresses a high-order, multivariate polynomial for conditional data generation.

Original: Importantly, MVP treats both discrete and continuous conditional variables in a uniﬁed way.
Revised: Importantly, MVP treats both discrete and continuous conditional variables in a uniﬁed way.

Revised: • We offer an in-depth relationship with state-of-the-art works, such as SPADE (Park et al., 2019), that can be interpreted as polynomial expansions.

Revised: We believe this perspective better explains the success of such architectures and offers a new direction for their extension.

Original: • MVP is trained on four different datasets for both class-conditional generation and imageto-image translation tasks.
Revised: • MVP is trained on eight different datasets for both class-conditional generation and imageto-image translation tasks.

Original: The trained models rely on both input variables, i.e., they do not ignore the noise vector.
Revised: The trained models rely on both input variables, i.e., they do not ignore the noise vector.

Original: • To illustrate the expressivity of the model, we also experiment with generators that do not use activation functions between the layers.
Revised: • To illustrate the expressivity of the model, we also experiment with generators that do not use activation functions between the layers.

Original: We verify that MVP can synthesize realistic images even in the absence of activation functions between the layers.
Revised: We verify that MVP can synthesize realistic images even in the absence of activation functions between the layers.

Original: The source code of MVP will be published upon the acceptance of the paper.
Revised: The source code of MVP will be published upon the acceptance of the paper.

Original:  R ELATED WORK
Revised:  R ELATED WORK

Original: The literature on conditional data generation is vast; dedicated surveys per task (Agnese et al., 2019;
Revised: The literature on conditional data generation is vast; dedicated surveys per task (Agnese et al., 2019;

Original: Wu et al., 2017b) can be found for the interested reader.
Revised: Wu et al., 2017b) can be found for the interested reader.

Original: Below, we review representative works in conditional generation and then we summarize the recent progress in multiplicative interactions.
Revised: Below, we review representative works in conditional generation and then we summarize the recent progress in multiplicative interactions.

Original: 2. C ONDITIONAL GENERATIVE MODELS
Revised: 2. C ONDITIONAL GENERATIVE MODELS

Original: The challenging nature of image/video generation has led to a proliferation of conditional models.
Revised: The challenging nature of image/video generation has led to a proliferation of conditional models.

Original: Although cGAN (Mirza & Osindero, 2014) is a general framework, since then the methods developed for conditional generation differ substantially depending on the type of conditional data.
Revised: Although cGAN (Mirza & Osindero, 2014) is a general framework, since then the methods developed for conditional generation differ substantially depending on the type of conditional data.

Original: We present below representative works of the two categories, i.e., discrete and continuous conditional data, and their combination.
Revised: We present below representative works of the two categories, i.e., discrete and continuous conditional data, and their combination.

Original: Discrete conditional variable :
Revised: Discrete conditional variable :

Original: This is most frequently used for class-conditional generation (Miyato et al., 2018;
Revised: This is most frequently used for class-conditional generation (Miyato et al., 2018;

Original: Brock et al., 2019; Kaneko et al., 2019).
Revised: Brock et al., 2019; Kaneko et al., 2019).

Original: Conditional normalization (Dumoulin et al., 2017; De Vries et al., 2017) techniques have been popular in the case of discrete conditional input, e.g., in generation of natural scenes images (Miyato et al., 2018;
Revised: Conditional normalization (Dumoulin et al., 2017; De Vries et al., 2017) techniques have been popular in the case of discrete conditional input, e.g., in generation of natural scenes images (Miyato et al., 2018;

Original: Brock et al., 2019).
Revised: Brock et al., 2019).

Original: Conditional normalization cannot trivially generalize to a continuous conditional variable.
Revised: Conditional normalization cannot trivially generalize to a continuous conditional variable.

Original: In AC-GAN (Odena et al., 2017), they concatenate the class labels with the noise; however, their model does not scale well (i.e., they train one model per 10 classes).
Revised: In AC-GAN (Odena et al., 2017), they concatenate the class labels with the noise; however, their model does not scale well (i.e., they train one model per 10 classes).

Original: The aforementioned methods cannot be trivially used or modiﬁed for continuous conditional input.
Revised: The aforementioned methods cannot be trivially used or modiﬁed for continuous conditional input.

Original: Text-to-image generation models (Qiao et al., 2019; Li et al., 2019; Zhang et al., 2018; Xu et al., 2018) use a specialized branch to embed the text labels.
Revised: Text-to-image generation models (Qiao et al., 2019; Li et al., 2019; Zhang et al., 2018; Xu et al., 2018) use a specialized branch to embed the text labels.

Original: Continuous conditional variable :
Revised: Continuous conditional variable :

Original: The inﬂuential work of pix2pix (Isola et al., 2017) has become the reference point for continuous conditional input.
Revised: The inﬂuential work of pix2pix (Isola et al., 2017) has become the reference point for continuous conditional input.

Original: The conditional input is embedded in a lowdimensional space (with an encoder), and then mapped to a high-dimensional output (through a decoder).
Revised: The conditional input is embedded in a lowdimensional space (with an encoder), and then mapped to a high-dimensional output (through a decoder).

Original: The framework has been widely used for inverse tasks (Ledig et al., 2017; Pathak et al., 2016; Wu et al., 2017a; Iizuka et al., 2017; Huang et al., 2017; Yu et al., 2018a; Grm et al., 2019; Xie et al., 2018; Yan & Wang, 2017), conditional pose generation (Ma et al., 2017; Siarohin et al., 2018;
Revised: The framework has been widely used for inverse tasks (Ledig et al., 2017; Pathak et al., ; Wu et al., 2017a; Iizuka et al., 2017; Huang et al., 2017; Yu et al., 2018a; Grm et al., 2019; Xie et al., 2018; Yan & Wang, 2017), conditional pose generation (Ma et al., 2017; Siarohin et al., 2018;

Original: Liang et al., 2019), representation learning (Tran et al., 2017), conditional video generation (Wang et al., 2018a), generation from semantic labels (Wang et al., 2018b), image blending (Wu et al., 2019;
Revised: Liang et al., 2019), representation learning (Tran et al., 2017), conditional video generation (Wang et al., 2018a), generation from semantic labels (Wang et al., 2018b), image blending (Wu et al., 2019;

Original: Zhan et al., 2019).
Revised: Zhan et al., 2019).

Original: We recognize two major drawbacks in the aforementioned methods: a) they cannot be easily adapted for discrete conditional input, b) they learn a deterministic mapping, i.e., the noise is typically ignored.
Revised: We recognize two major drawbacks in the aforementioned methods: a) they cannot be easily adapted for discrete conditional input, b) they learn a deterministic mapping, i.e., the noise is typically ignored.

Original: However, in many real applications, such as inverse tasks, the mapping is not one-to-one; there are multiple plausible outputs for every conditional input.
Revised: However, in many real applications, such as inverse tasks, the mapping is not one-to-one; there are multiple plausible outputs for every conditional input.

Original: The auxiliary losses used in such works, e.g., (cid:96) 1 loss (Isola et al., 2017), perceptual loss (Ledig et al., 2017), are an additional drawback.
Revised: The auxiliary losses used in such works, e.g., (cid:96) 1 loss (Isola et al., 2017), perceptual loss (Ledig et al., 2017), are an additional drawback.

Original: Those losses both add hyper-parameters that require tuning and are domain-speciﬁc, thus it is challenging to transfer them to different domains or even different datasets.
Revised: Those losses both add hyper-parameters that require tuning and are domain-speciﬁc, thus it is challenging to transfer them to different domains or even different datasets.

Original: On the contrary, in our experiments, we do not use any additional loss.
Revised: On the contrary, in our experiments, we do not use any additional loss.

Original: The works of Zhu et al. (2017b); Huang et al. ; Alami Mejjati et al. (2018); Gonzalez-Garcia et al. ; Lee et al.
Revised: Discrete and continuous conditional variables : Few works combine both discrete and continuous conditional inputs (Yu et al., 2018b; Xu et al., 2017; Lu et al., 2018).

Original: ; Choi et al.

Original: (2020) propose well-designed architectures that include multiple generators and discriminators in order to synthesize diverse images.

Original: Even though they demonstrate impressive results, the complicated training schemes (and additional network modules) make such works harder to be adapted to different tasks.

Original: Discrete and continuous conditional variables : Few works combine both discrete and continuous conditional inputs (Yu et al., 2018b; Xu et al., 2017; Lu et al., 2018).

Original: However, these methods include signiﬁcant engineering (e.g., multiple discriminators (Xu et al., 2017), auxiliary losses), while often the generator learns to ignore the noise (similarly to the continuous conditional input).
Revised: However, these methods include signiﬁcant engineering (e.g., multiple discriminators (Xu et al., 2017), auxiliary losses), while often the generator learns to ignore the noise (similarly to the continuous conditional input).

Original: Antipov et al.
Revised: Antipov et al.

Original: (2017) design a generator for face aging.
Revised: (2017) design a generator for face aging.

Original: The generator combines continuous with discrete variables (age classes), however there is no Gaussian noise utilized, i.e., a deterministic transformation is learned for each input face.
Revised: The generator combines continuous with discrete variables (age classes), however there is no Gaussian noise utilized, i.e., a deterministic transformation is learned for each input face.

Original: InfoGAN (Chen et al., 2016) includes both discrete and continuous conditional variables.
Revised: InfoGAN (Chen et al., 2016) includes both discrete and continuous conditional variables.

Original: However, the authors explicitly mention that additional losses are required, otherwise the generator is ‘free to ignore’ the additional variables.
Revised: However, the authors explicitly mention that additional losses are required, otherwise the generator is ‘free to ignore’ the additional variables.

Original: The idea of Li et al.
Revised: The idea of Li et al.

Original: (2020) is most closely related to our work.
Revised: (2020) is most closely related to our work.

Original: They introduce a unifying framework for paired (Isola et al., 2017) and unpaired (Zhu et al., 2017a) learning.
Revised: They introduce a unifying framework for paired (Isola et al., 2017) and unpaired (Zhu et al., 2017a) learning.

Original: However, their framework assumes a continuous conditional input, while ours can handle discrete conditional input (e.g., class labels).
Revised: However, their framework assumes a continuous conditional input, while ours can handle discrete conditional input (e.g., class labels).

Original: In addition, their method requires a pre-trained teacher generator, while ours consists of a single generator trained end-to-end.
Revised: In addition, their method requires a pre-trained teacher generator, while ours consists of a single generator trained end-to-end.

Revised: Diverse data generation : Conditional image generation often suffers from deterministic mappings, i.e., the noise variable has often negligible or negative impact in the generator (Zhu et al., 2017b;

Revised: Isola et al., 2017).

Revised: This has been tackled in the literature with additional loss terms and/or auxiliary network modules.

Revised: A discussion of representative methods that tackle diverse generation is deferred to sec.

Original: 2. M ULTIPLICATIVE INTERACTIONS
Revised: I in the Appendix. In Table 1 the differences of the core techniques are summarized. Even though diverse generation is a signiﬁcant task, we advocate that learning a generator does not ignore the input variables can be achieved without such additional loss terms. We highlight that diverse generation is a byproduct of MVP and not our main goal.

Revised: Particularly, we believe that diverse images can be synthesized because the higher-order correlations of the input variables are captured effectively the proposed method.

Revised: [Table]

Revised: 2. M ULTIPLICATIVE INTERACTIONS

Original: Multiplicative connections have long been adopted in computer vision and machine learning (Shin &
Revised: Multiplicative connections have long been adopted in computer vision and machine learning (Shin &

Original: Ghosh, 1991; Hochreiter & Schmidhuber, 1997; Bahdanau et al., 2015).
Revised: Ghosh, 1991; Hochreiter & Schmidhuber, 1997; Bahdanau et al., 2015).

Original: The idea is to combine the inputs through elementwise products or other diagonal forms.
Revised: The idea is to combine the inputs through elementwise products or other diagonal forms.

Original: Even though multiplicative connections have successfully been applied to different tasks, until recently there was no comprehensive study of their expressivity versus the standard feedforward networks.
Revised: Even though multiplicative connections have successfully been applied to different tasks, until recently there was no comprehensive study of their expressivity versus the standard feedforward networks.

Original: Jayakumar et al.
Revised: Jayakumar et al.

Original: (2020) include the proof that second order multiplicative operators can represent a greater class of functions than classic feed-forward networks.
Revised: (2020) include the proof that second order multiplicative operators can represent a greater class of functions than classic feed-forward networks.

Original: Even though we capitalize on the theoretical argument, our framework can express any higher-order interactions while the framework of Jayakumar et al.
Revised: Even though we capitalize on the theoretical argument, our framework can express any higher-order interactions while the framework of Jayakumar et al.

Original: (2020) is limited to second order interactions. 
Revised: (2020) is limited to second order interactions. [Table]

Revised: (cid:88)(cid:88)(cid:88) (cid:88)(cid:88)(cid:88) (cid:88)(cid:88)(cid:88)

Original: Higher-order interactions have been studied in the tensor-related literature (Kolda & Bader, 2009;
Revised: Higher-order interactions have been studied in the tensor-related literature (Kolda & Bader, 2009;

Original: Debals & De Lathauwer, 2017).
Revised: Debals & De Lathauwer, 2017).

Original: However, their adaptation in modern deep architectures has been slower.
Revised: However, their adaptation in modern deep architectures has been slower.

Original: Chrysos et al.
Revised: Chrysos et al.

Original: (2020) propose high-order polynomial for mapping the input z to the output x “ G p z q .
Revised: (2020) propose high-order polynomial for mapping the input z to the output x “ G p z q .

Original: Their framework is limited to a single input variable and cannot handle the multivariate cases that are the focus of this work.
Revised: Π -Netfocuses on a single input variable and cannot handle the multivariate cases that are the focus of this work.

Original: SPADE (Park et al., 2019) can be interpreted asa polynomial expansion with respect to the conditional variable, i.e., a semantic layout in their case. Capturing higher-order correlations with respect to one of the variables (and not the rest) restricts the expressivity of the model.
Revised: Three additional works that can be thought of as polynomial expansions are

Original: Further differences of MVP from Chrysos et al.
Revised: Karras et al.

Original: and Park et al.
Revised: ; Park et al.

Original: (2019) are deferred to sec.
Revised: ; Chen et al. The three works were originally introduced as (conditional) normalization variants, but we attribute their improvements in the expressiveness of their polynomial expansions. Under the polynomial expansion perspective, they can be expressed as special cases of the proposed MVP.

Original: E and sec.
Revised: A detailed discussion is conducted in sec.

Original: F respectively
Revised: F in the Appendix.

Original: [Table]
Revised: We believe that the proposed framework offers a direction to further extend the results of such works, e.g., by allowing more than one conditional variables.

Original:  M ETHOD
Revised:  M ETHOD

Original: [Table] The framework for a multivariate polynomial with a two-variable input is introduced (sec. 3.1).
Revised: The framework for a multivariate polynomial with a two-variable input is introduced (sec. 3.1).

Original: The derivation, further intuition and additional models are deferred to the Appendix (sec. B).
Revised: The derivation, further intuition and additional models are deferred to the Appendix (sec. B).

Original: The crucial technical details, including the stability of the polynomial [Table] are developed in sec.
Revised: The crucial technical details, including the stability of the polynomial, are developed in sec.

Original: We emphasize that a multivari- ate polynomial can approximate any function (Stone, 1948; Nikol’skii, 2013), i.e., a multivariate polynomial is a universal approximator. 
Revised: We emphasize that a multivariate polynomial can approximate any function (Stone, 1948; Nikol’skii, 2013), i.e., a multivariate polynomial is a universal approximator. [Table]

Original: Notation :Tensors/matrices/vectors are symbolized by calligraphic/uppercase/lowercase boldface letters e.g., W , W , w .
Revised: Notation :Tensors/matrices/vectors are symbolized by calligraphic/uppercase/lowercase boldface letters e.g., W , W , w .

Original: The mode- m vector product of W (of order M ) with a vector u P R I m is W ˆ m u and results in a tensor of order M ´ 1 .
Revised: The mode- m vector product of W (of order M ) with a vector u P R I m is W ˆ m u and results in a tensor of order M ´ 1 .

Original: We assume that ś bi “ a x i “ 1 when a ą b .
Revised: We assume that ś bi “ a x i “ 1 when a ą b .

Original: The core symbols are summarized in Table 1, while a detailed tensor notation is deferred to the Appendix (sec. B.1).
Revised: The core symbols are summarized in Table 3, while a detailed tensor notation is deferred to the Appendix (sec. B.1).

Original: 3. T WO INPUT VARIABLES
Revised: 3. T WO INPUT VARIABLES

Original: Given two input variables 

Revised: Given two input variables

Original: z
Revised: z

Original: I , z II P K d where K Ď R or K Ď N , the goal is to learn a function
Revised: I , z II P K d where K Ď R or K Ď N , the goal is to learn a function

Original: G : K d ˆ d Ñ R o that captures the higher-degree interactions between the elements of the two inputs.
Revised: G : K d ˆ d Ñ R o that captures the higher-degree interactions between the elements of the two inputs.

Original: We can learn such higher-degree interactions as polynomials of two input variables.
Revised: We can learn such higher-degree interactions as polynomials of two input variables.

Original: A polynomial of expansion order N P N with output x P R o has the form:
Revised: A polynomial of expansion order N P N with output x P R o has the form:

Original: [Equation]
Revised: [Equation]

Original: where β P R o and W r n,ρ s P R o ˆ ś nm “ 1 ˆ m d for n P r 1 , N s , ρ P r 1 , n ` 1 s are the learnable parameters.
Revised: where β P R o and W r n,ρ s P R o ˆ ś nm “ 1 ˆ m d for n P r 1 , N s , ρ P r 1 , n ` 1 s are the learnable parameters.

Original: The expansion depends on two (independent) variables, hence we use the n and ρ as auxiliary variables.
Revised: The expansion depends on two (independent) variables, hence we use the n and ρ as auxiliary variables.

Original: The two products of (1) do not overlap, i.e., the ﬁrst multiplies the modes r 2 , ρ s (of W r n,ρ s ) with z
Revised: The two products of (1) do not overlap, i.e., the ﬁrst multiplies the modes r 2 , ρ s (of W r n,ρ s ) with z

Original: I and the other multiplies the modes r ρ ` 1 , n ` 1 s with z II .
Revised: I and the other multiplies the modes r ρ ` 1 , n ` 1 s with z II .

Original: [Equation]
Revised: [Equation]

Original: ∼ ∼ 𝐵𝑀𝑊,… or
Revised: [Figure]

Original: Recursive relationship : The aforementioned derivation can be generalized to an arbitrary expansion order.
Revised: Recursive relationship : The aforementioned derivation can be generalized to an arbitrary expansion order.

Original: The recursive formula for an arbitrary order N P N is the following:
Revised: The recursive formula for an arbitrary order N P N is the following:

Original: [Equation]
Revised: [Equation]

Original: for n “ 2 , . . .
Revised: for n “ 2 , . . .

Original: , N with x 1 “ U T r 1 ,I s z
Revised: , N with x 1 “ U T r 1 ,I s z

Original: I ` U T r 1 ,II s z II and x “ Cx N ` β .
Revised: I ` U T r 1 ,II s z II and x “ Cx N ` β .

Original: The parameters C P
Revised: The parameters C P

Original: R o ˆ k , U r n,φ s P R d ˆ k for n “ 1 , . .
Revised: R o ˆ k , U r n,φ s P R d ˆ k for n “ 1 , . .

Original: . , N and φ “ t I, II u are learnable.
Revised: . , N and φ “ t I, II u are learnable.

Original: The intuition behind this model is the following: An embedding is initially found for each of the two input variables, then the two embeddings are added together and they are multiplied elementwise with the previous approximation.
Revised: The intuition behind this model is the following: An embedding is initially found for each of the two input variables, then the two embeddings are added together and they are multiplied elementwise with the previous approximation.

Original: The different embeddings for each of the input variables allows us to implement U r n,I s and U r n,II s with different constraints, e.g., U r n,I s to be a dense layer and U r n,II s to be a convolution.
Revised: The different embeddings for each of the input variables allows us to implement U r n,I s and U r n,II s with different constraints, e.g., U r n,I s to be a dense layer and U r n,II s to be a convolution.

Original: 3. M ODEL EXTENSIONS AND TECHNICAL DETAILS
Revised: 3. M ODEL EXTENSIONS AND TECHNICAL DETAILS

Original: There are three limitations in (2).
Revised: There are three limitations in (2).

Original: Those are the following: a) (2) describes a polynomial expansion of a two-variable input, b) each expansion order requires additional layers, c) high-order polynomials might suffer from unbounded values.
Revised: Those are the following: a) (2) describes a polynomial expansion of a two-variable input, b) each expansion order requires additional layers, c) high-order polynomials might suffer from unbounded values.

Original: Those limitations are addressed below.
Revised: Those limitations are addressed below.

Original: Our model can be readily extended beyond two-variable input; an extension with three-variable input is developed in sec.
Revised: Our model can be readily extended beyond two-variable input; an extension with three-variable input is developed in sec.

Original: C.
Revised: C.

Original: The pattern (for each order) is similar to the two-variable input: a) a different embedding is found for each input variable, b) the embeddings are added together, c) the result is multiplied elementwise with the representation of the previous order.
Revised: The pattern (for each order) is similar to the two-variable input: a) a different embedding is found for each input variable, b) the embeddings are added together, c) the result is multiplied elementwise with the representation of the previous order.

Original: The polynomial expansion of (2) requires Θ p N q layers for an N th order expansion.
Revised: The polynomial expansion of (2) requires Θ p N q layers for an N th order expansion.

Original: That is, each new order n of expansion requires new parameters U r n,I s and U r n,II s .
Revised: That is, each new order n of expansion requires new parameters U r n,I s and U r n,II s .

Original: However, the order of expansion can be increased without increasing the parameters substantially.
Revised: However, the order of expansion can be increased without increasing the parameters substantially.

Original: To that end, we can capitalize on the product of polynomials.
Revised: To that end, we can capitalize on the product of polynomials.

Original: Speciﬁcally, let N 1 be the order of expansion of the ﬁrst polynomial.
Revised: Speciﬁcally, let N 1 be the order of expansion of the ﬁrst polynomial.

Original: The output of the ﬁrst polynomial is fed into a second polynomial, which has expansion order of N 2 .
Revised: The output of the ﬁrst polynomial is fed into a second polynomial, which has expansion order of N 2 .

Original: Then, the output of the second polynomial will have an expansion order of N 1 ¨ N 2 .
Revised: Then, the output of the second polynomial will have an expansion order of N 1 ¨ N 2 .

Original: The product of polynomials can be used with arbitrary number of polynomials; it sufﬁces the output of the τ th polynomial to be the input to the p τ ` 1 q th polynomial.
Revised: The product of polynomials can be used with arbitrary number of polynomials; it sufﬁces the output of the τ th polynomial to be the input to the p τ ` 1 q th polynomial.

Original: For instance, if we assume a product of Φ P N polynomials, where each polynomial has an expansion order of two, then the polynomial expansion is of 2 Φ order.
Revised: For instance, if we assume a product of Φ P N polynomials, where each polynomial has an expansion order of two, then the polynomial expansion is of 2 Φ order.

Original: In other words, we need Θ p log 2 p N qq layers to achieve an N th order expansion.
Revised: In other words, we need Θ p log 2 p N qq layers to achieve an N th order expansion.

Original: In algebra, higher-order polynomials are unbounded and can thus suffer from instability for large values.
Revised: In algebra, higher-order polynomials are unbounded and can thus suffer from instability for large values.

Original: To avoid such instability, we take the following three steps: a) MVP samples the noise vector from the uniform distribution, i.e., from the bounded interval of r´ 1 , 1 s , b) a hyperbolic tangent is used in the output of the generator as a normalization, i.e., it constrains the outputs in the bounded interval of r´ 1 , 1 s , c) batch normalization (Ioffe & Szegedy, 2015) is used to convert the representations to zero-mean.
Revised: To avoid such instability, we take the following three steps: a) MVP samples the noise vector from the uniform distribution, i.e., from the bounded interval of r´ 1 , 1 s , b) a hyperbolic tangent is used in the output of the generator as a normalization, i.e., it constrains the outputs in the bounded interval of r´ 1 , 1 s , c) batch normalization (Ioffe & Szegedy, 2015) is used to convert the representations to zero-mean.

Original: We emphasize that in GANs the hyperbolic tangent is the default activation function in the output of the generator, hence it is not an additional requirement of our method.
Revised: We emphasize that in GANs the hyperbolic tangent is the default activation function in the output of the generator, hence it is not an additional requirement of our method.

Original: Additionally, in our preliminary experiments, the uniform distribution can be changed for a Gaussian distribution without any instability.
Revised: Additionally, in our preliminary experiments, the uniform distribution can be changed for a Gaussian distribution without any instability.

Original: A theoretical analysis on the bounds of such multivariate polynomials would be an interesting subject for future work.
Revised: A theoretical analysis on the bounds of such multivariate polynomials would be an interesting subject for future work.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: The proposed MVP is empirically evaluated in three settings: a) a class-conditional generation, i.e., with discrete conditional input, b) an image-to-image translation, i.e., with continuous conditional input, c) a mixed conditional setting with two conditional variables.
Revised: The proposed MVP is empirically evaluated in three settings: a) a class-conditional generation, i.e., with discrete conditional input, b) an image-to-image translation, i.e., with continuous conditional input, c) a mixed conditional setting with two conditional variables.

Original: The goal is to showcase how
Revised: The goal is to showcase how

Original: MVP can be used with both discrete and continuous conditional inputs.
Revised: MVP can be used with both discrete and continuous conditional inputs.

Original: Even though architectures specialized for a single task (e.g., Ledig et al.
Revised: Even though architectures specialized for a single task (e.g., Ledig et al.

Original: (2017)) perform well in that task, their well-selected inductive biases (e.g., perceptual or (cid:96) 1 loss) do not generalize well in other domains or different conditional inputs.
Revised: (2017)) perform well in that task, their well-selected inductive biases (e.g., perceptual or (cid:96) 1 loss) do not generalize well in other domains or different conditional inputs.

Original: Hence, our goal is not to demonstrate state-of-the-art results in speciﬁc tasks, but rather to propose one generic formulation.
Revised: Hence, our goal is not to demonstrate state-of-the-art results in speciﬁc tasks, but rather to propose one generic formulation.

Original: Further experiments (e.g., class-conditional generation with SVHN or MNIST to SVHN translation; sec H), the details on the datasets and the evaluation metrics (sec.
Revised: Further experiments (e.g., class-conditional generation with SVHN or MNIST to SVHN translation; sec H), the details on the datasets and the evaluation metrics (sec.

Original: G) are deferred to the Appendix.
Revised: G) are deferred to the Appendix.

Original: Throughout the experimental section, we reserve the symbol z II for the conditional input (e.g., a class label).
Revised: Throughout the experimental section, we reserve the symbol z II for the conditional input (e.g., a class label).

Original: Our framework, e.g., (2), does not include any activation functions.
Revised: Our framework, e.g., (2), does not include any activation functions.

Original: To verify the expressivity of our framework, we maintain the same setting for the majority of the experiments below.
Revised: To verify the expressivity of our framework, we maintain the same setting for the majority of the experiments below.

Original: Particularly, the generator does not have activation functions between the layers; there is only a hyperbolic tangent in the output space for normalization.
Revised: Particularly, the generator does not have activation functions between the layers; there is only a hyperbolic tangent in the output space for normalization.

Original: Training a generator without activation functions between the layers also emerged in Π -Net (Chrysos et al., 2020), where the authors demonstrate the challenges in such framework.
Revised: Training a generator without activation functions between the layers also emerged in Π -Net (Chrysos et al., 2020), where the authors demonstrate the challenges in such framework.

Original: However, we conduct one experiment using a strong baseline with activation functions.
Revised: However, we conduct one experiment using a strong baseline with activation functions.

Original: That is, a comparison with SNGAN (Miyato & Koyama, 2018) in class-conditional generation is performed (sec. 4.1).
Revised: That is, a comparison with SNGAN (Miyato & Koyama, 2018) in class-conditional generation is performed (sec. 4.1).

Original: Baselines: ‘ Π -Net-SICONC’ implements a polynomial expansion of a single variable, i.e., by concatenating all the input variables.
Revised: Baselines: ‘ Π -Net-SICONC’ implements a polynomial expansion of a single variable, i.e., by concatenating all the input variables.

Original: ‘SPADE’ implements a polynomial expansion with respect to the conditional variable.
Revised: ‘SPADE’ implements a polynomial expansion with respect to the conditional variable.

Original: Also, ‘GAN-CONC’ and ‘GAN-ADD’ are added as baselines, where we replace the Hadamard products with concatenation and addition respectively.
Revised: Also, ‘GAN-CONC’ and ‘GAN-ADD’ are added as baselines, where we replace the Hadamard products with concatenation and addition respectively.

Original: An abstract schematic of the differences between the compared polynomial methods is depicted in Fig.
Revised: An abstract schematic of the differences between the compared polynomial methods is depicted in Fig.

Original: 6, while a detailed description of all methods is deferred to sec.
Revised: 6, while a detailed description of all methods is deferred to sec.

Original: G. Each experiment is conducted ﬁve times and the mean and the standard deviation are reported.
Revised: G. Each experiment is conducted ﬁve times and the mean and the standard deviation are reported.

Original: 4. C LASS - CONDITIONAL GENERATION
Revised: 4. C LASS - CONDITIONAL GENERATION

Original: The ﬁrst experiment is on class-conditional generation, where the conditional input is a class label in the form of one-hot vector.
Revised: The ﬁrst experiment is on class-conditional generation, where the conditional input is a class label in the form of one-hot vector.

Original: Two types of networks are utilized: a) a resnet-based generator (SNGAN), b) a polynomial generator ( Π -Net) based on Chrysos et al.
Revised: Two types of networks are utilized: a) a resnet-based generator (SNGAN), b) a polynomial generator ( Π -Net) based on Chrysos et al.

Original: The former network has exhibited strong performance the last few years, while the latter bears resemblance to the formulation we propose in this work.
Revised: The former network has exhibited strong performance the last few years, while the latter bears resemblance to the formulation we propose in this work.

Original: [Table]
Revised: [Table]

Original: Resnet-based generator: The experiment is conducted by augmenting the resnet-based generator of SNGAN.
Revised: Resnet-based generator: The experiment is conducted by augmenting the resnet-based generator of SNGAN.

Original: The quantitative results are in Table 2 and synthesized samples are illustrated in Fig.
Revised: The quantitative results are in Table 4 and synthesized samples are illustrated in Fig.

Original: 2(a).
Revised: 2(a).

Original: SNGAN-MVP improves upon all the baselines in both the Inception score (IS) (Salimans et al., 2016) and the FID (Heusel et al., 2017).
Revised: SNGAN-MVP improves upon all the baselines in both the Inception score (IS) (Salimans et al., 2016) and the FID (Heusel et al., 2017).

Original: The proposed formulation enables inter-class interpolations.
Revised: The proposed formulation enables inter-class interpolations.

Original: That is, the noise z
Revised: That is, the noise z

Original: I is ﬁxed, while the class z II is interpolated.
Revised: I is ﬁxed, while the class z II is interpolated.

Original: In Fig. 2(b) and Fig. 2(c), intra-class and inter-class linear interpolations are illustrated respectively.
Revised: In Fig. 2(b) and Fig. 2(c), intra-class and inter-class linear interpolations are illustrated respectively.

Original: Both the quantitative and the qualitative results exhibit the effectiveness of our framework.
Revised: Both the quantitative and the qualitative results exhibit the effectiveness of our framework.

Original: [Figure]
Revised: [Figure]

Revised: source to the target, (c) inter-class linear interpolation.

Revised: In inter-class interpolation, the class labels of the leftmost and rightmost images are one-hot vectors, while the rest are interpolated in-between; the resulting images are visualized.

Revised: In all three cases, MVP synthesizes realistic images.

Original:  Π -Net-based generator: A product of polynomials, based on Π -Net, is selected as the baseline architecture for the generator.
Revised: [Table] Π -Net-based generator: A product of polynomials, based on Π -Net, is selected as the baseline architecture for the generator.

Original: Π -Net has conditional batch normalization (CBN) in the generator, while in the rest compared methods CBN is replaced by batch normalization.
Revised: Π -Net has conditional batch normalization (CBN) in the generator, while in the rest compared methods CBN is replaced by batch normalization.

Original: The results in CIFAR are summarized in Table 3 (left), where MVP outperforms all the baselines by a large margin.An [Table]
Revised: The results in CIFAR are summarized in Table 5 (left), where MVP outperforms all the baselines by a large margin.

Original:  additional experiment is performed in Cars196 that has 196 classes.
Revised: An additional experiment is performed in Cars196 that has 196 classes.

Original: The results in Table 3 (right) depict a substantial improvement over the all the baselines ( 53 . 9% reduction over the best-performing baseline).
Revised: The results in Table 5 (right) depict a substantial improvement over the all the baselines ( 53 . 9% reduction over the best-performing baseline).

Original: We should note that the baseline was not built for conditional generation, however we have done our best effort to optimize the respective hyper-parameters.
Revised: We should note that the baseline was not built for conditional generation, however we have done our best effort to optimize the respective hyper-parameters.

Original: We hypothesize that the improvement arises because of the correlations of the classes.
Revised: We hypothesize that the improvement arises because of the correlations of the classes.

Original: That is, the 196 classes might be correlated (e.g., the SUV cars of different carmakers share several patterns).
Revised: That is, the 196 classes might be correlated (e.g., the SUV cars of different carmakers share several patterns).

Original: Such correlations are captured by our framework, while they might be missed when learning different normalization statistics per class.
Revised: Such correlations are captured by our framework, while they might be missed when learning different normalization statistics per class.

Original: Overall, MVP synthesizes plausible images (Fig. 11) even in the absence of activation functions.
Revised: Overall, MVP synthesizes plausible images (Fig. 11) even in the absence of activation functions.

Original: 4. C ONTINUOUS CONDITIONAL INPUT
Revised: 4. C ONTINUOUS CONDITIONAL INPUT

Original: The performance of MVP is scrutinized in tasks with continuous conditional input, e.g., superresolution.
Revised: The performance of MVP is scrutinized in tasks with continuous conditional input, e.g., superresolution.

Original: The conditional input z II is an input image, e.g., a low-resolution sample or a corrupted sample.
Revised: The conditional input z II is an input image, e.g., a low-resolution sample or a corrupted sample.

Original: Even though the core architecture remains the same, a single change is made in the structure of the discriminator:
Revised: Even though the core architecture remains the same, a single change is made in the structure of the discriminator:

Original: Motivated by (Miyato & Koyama, 2018), we include an elementwise product of z II with the real/fake image in the discriminator.
Revised: Motivated by (Miyato & Koyama, 2018), we include an elementwise product of z II with the real/fake image in the discriminator.

Original: This stabilizes the training and improves the results.
Revised: This stabilizes the training and improves the results.

Original: A wealth of literature is available on such continuous conditional inputs (sec. 2.1), however we select the challenging setting of using a generator without activation functions between the layers. 
Revised: A wealth of literature is available on such continuous conditional inputs (sec. 2.1), however we select the challenging setting of using a generator without activation functions between the layers. in sec. G), vastly improves the original SPADE.

Revised: The full two-variable model, i.e., MVP, outperforms the compared methods.

Original: [Table]
Revised: [Table]

Original: The experiments are performed in (a) super-resolution, (b) block-inpainting.
Revised: The experiments are performed in (a) super-resolution, (b) block-inpainting.

Original: Super-resolution assumes a low-resolution image is available, while in block inpainting, a (rectangular) part of the image is missing.
Revised: Super-resolution assumes a low-resolution image is available, while in block inpainting, a (rectangular) part of the image is missing.

Original: The two tasks belong in the broader category of ‘inverse tasks’, and they are signiﬁcant both for academic reasons but also for commercial reasons (Sood et al., 2018;
Revised: The two tasks belong in the broader category of ‘inverse tasks’, and they are signiﬁcant both for academic reasons but also for commercial reasons (Sood et al., 2018;

Original: You et al., 2019).
Revised: You et al., 2019).

Original: Such inverse tasks are underdetermined; each input image corresponds to several plausible output images.
Revised: Such inverse tasks are underdetermined; each input image corresponds to several plausible output images.

Original: The FID scores in Cars196 for the task of super-resolution are reported in Table 4.
Revised: The FID scores in Cars196 for the task of super-resolution are reported in Table 6.

Original: In super-resolution 16 ˆ , z II has 48 dimensions, while in super-resolution 8 ˆ , z II has 192 dimensions.
Revised: In super-resolution 16 ˆ , z II has 48 dimensions, while in super-resolution 8 ˆ , z II has 192 dimensions.

Original: Notice that the performance of Π -Net-SICONC deteriorates substantially when the dimensionality of the conditional variable increases.
Revised: Notice that the performance of Π -Net-SICONC deteriorates substantially when the dimensionality of the conditional variable increases.

Original: That validates our intuition about the concatenation in the input of the generator (sec. E).
Revised: That validates our intuition about the concatenation in the input of the generator (sec. E).

Original: We also report the SPADE-MVP, which captures higher-order correlations with respect[Figure] to the ﬁrst variable as well (further details in sec. G).
Revised: We also report the SPADE-MVP, which captures higher-order correlations with respect to the ﬁrst variable as well (further details in sec. G).

Original: The proposed SPADE-MVP outperforms the original SPADE, however it cannot outperform the full two-variable model, i.e., MVP.
Revised: The proposed SPADE-MVP outperforms the original SPADE, however it cannot outperform the full two-variable model, i.e., MVP.

Original: MVP maintains outperforms all baselines by a large margin.
Revised: MVP maintains outperforms all baselines by a large margin.

Original: The qualitative results on (a) super-resolution 8 ˆ on CelebA, (b) super-resolution 8 ˆ on Cars196, (c) super-resolution 16 ˆ on Cars196 are illustrated in Fig.
Revised: The qualitative results on (a) super-resolution 8 ˆ on CelebA, (b) super-resolution 8 ˆ on Cars196, (c) super-resolution 16 ˆ on Cars196 are illustrated in Fig.

Original: Similarly the qualitative results on block-inpainting are visualized in Fig.
Revised: Similarly the qualitative results on block-inpainting are visualized in Fig.

Original: For each conditional image, different noise vectors z
Revised: For each conditional image, different noise vectors z

Original: I are sampled.
Revised: I are sampled.

Original: Notice that the corresponding synthesized images differ in the ﬁne-details.
Revised: Notice that the corresponding synthesized images differ in the ﬁne-details.

Original: For instance, changes in the mouth region, the car type or position and even background changes are observed.
Revised: For instance, changes in the mouth region, the car type or position and even background changes are observed.

Original: Thus, MVP results in high-resolution images that i) correspond to the conditional input, ii) vary in ﬁne-details.
Revised: Thus, MVP results in high-resolution images that i) correspond to the conditional input, ii) vary in ﬁne-details.

Original: Similar variation has emerged even when the source and the target domains differ substantially, e.g., in the translation of MNIST digits to SVHN digits (sec.
Revised: Similar variation has emerged even when the source and the target domains differ substantially, e.g., in the translation of MNIST digits to SVHN digits (sec.

Original: H.3).
Revised: H.3).

Original: We should mention that regularization techniques have been proposed speciﬁcally for image-to-image translation, e.g., 
Revised: We should mention that regularization techniques have been proposed speciﬁcally for image-to-image translation, e.g., [Figure]

Original: Yang et al.
Revised: Yang et al.

Original: ; Lee et al.
Revised: ; Lee et al.

Original: However, such works utilize additional losses and even require additional networks for training, which makes the training more computationally heavy and more sensitive to design choices.
Revised: However, such works utilize additional losses and even require additional networks for training, which makes the training more computationally heavy and more sensitive to design choices.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: The topic of conditional data generation is the focus of this work.
Revised: The topic of conditional data generation is the focus of this work.

Original: A multivariate polynomial model, called MVP, is introduced.
Revised: A multivariate polynomial model, called MVP, is introduced.

Original: MVP approximates a function G p
Revised: MVP approximates a function G p

Original: z I , z II q with inputs z I (e.g., sample from a Gaussian distribution) and z II (e.g., class or low-resolution image).
Revised: z I , z II q with inputs z I (e.g., sample from a Gaussian distribution) and z II (e.g., class or low-resolution image).

Original: MVP resorts to multivariate polynomials with arbitrary conditional inputs, which capture high-order correlations of the inputs.
Revised: MVP resorts to multivariate polynomials with arbitrary conditional inputs, which capture high-order correlations of the inputs.

Original: The empirical evaluation conﬁrms that our framework can synthesize realistic images in both classconditional generation (trained on CIFAR10 and Cars196), and image-to-image translation (both super-resolution and block-inpainting).
Revised: The empirical evaluation conﬁrms that our framework can synthesize realistic images in both class-conditional generation (trained on CIFAR10, Cars196 and SVHN), attribute-guided generation and image-to-image translation (i.e., super-resolution, block-inpainting, edges-to-shoes, edges-to-handbag, MNIST-to-SVHN).

Original: We also showcase that it can be extended to three-variable input with class-conditional super-resolution.
Revised: We also showcase that it can be extended to three-variable input with class-conditional super-resolution.

Original: In addition to conditional data generation, the proposed framework can be used in tasks requiring fusion of different types of variables.
Revised: In addition to conditional data generation, the proposed framework can be used in tasks requiring fusion of different types of variables.

Original: A S UMMARY OF SECTIONS IN THE A PPENDIX
Revised: A S UMMARY OF SECTIONS IN THE A PPENDIX

Original: In the following sections, further details and derivations are provided to elaborate the details of the
Revised: In the following sections, further details and derivations are provided to elaborate the details of the

Original: MVP.
Revised: MVP.

Original: Speciﬁcally, in sec.
Revised: Speciﬁcally, in sec.

Original: B the decomposition and related details on the method are developed.
Revised: B the decomposition and related details on the method are developed.

Original: The extension of our method beyond two-input variables is studied in sec.
Revised: The extension of our method beyond two-input variables is studied in sec.

Original: C. A method frequently used in the literature for fusing information is concatenation; we analyze how concatenation captures only additive and not more complex correlations (e.g., multiplicative) in sec.
Revised: C. A method frequently used in the literature for fusing information is concatenation; we analyze how concatenation captures only additive and not more complex correlations (e.g., multiplicative) in sec.

Original: D. The two methods that are closest to our work are Π -Net (Chrysos et al., 2020) and SPADE (Park et al., 2019). An analysis of the differences from each methods are developed in sec.
Revised: D. The differences from Π -Net (Chrysos et al., 2020) is explored in sec.

Original: E and sec.
Revised: E. In sec.

Original: F respectively.
Revised: F, some recent (conditional) data generation methods are cast into the polynomial neural network framework and their differences from the proposed framework are analyzed.

Original: The experimental details including the evaluation metrics and details on the baselines are developed in sec.
Revised: The experimental details including the evaluation metrics and details on the baselines are developed in sec.

Original: G.
Revised: G.

Original: In sec.
Revised: In sec.

Original: H, additional experimental results are included.
Revised: H, additional experimental results are included.

Revised: Lastly, the differences from works that perform diverse generation are explored in sec.

Revised: I.

Original: B M ETHOD DERIVATIONS
Revised: B M ETHOD DERIVATIONS

Original: In this section, we expand on the method details, including the scalar output case or the notation.
Revised: In this section, we expand on the method details, including the scalar output case or the notation.

Original: Speciﬁcally, a more detailed notation is determined in sec.
Revised: Speciﬁcally, a more detailed notation is determined in sec.

Original: B.1; the scalar output case is analyzed in sec.
Revised: B.1; the scalar output case is analyzed in sec.

Original: B.2.
Revised: B.2.

Original: In sec.
Revised: In sec.

Original: B.3 a second order expansion is assumed to illustrate the connection between the polynomial expansion and the recursive formula.
Revised: B.3 a second order expansion is assumed to illustrate the connection between the polynomial expansion and the recursive formula.

Original: Sequentially, we derive an alternative model with different factor sharing.
Revised: Sequentially, we derive an alternative model with different factor sharing.

Original: This model, called Nested-MVP, has a nested factor sharing format (sec. B.4).
Revised: This model, called Nested-MVP, has a nested factor sharing format (sec. B.4).

Original: B. N OTATION
Revised: B. N OTATION

Original: Our derivations rely on tensors (i.e., multidimensional equivalent of matrices) and (tensor) products.
Revised: Our derivations rely on tensors (i.e., multidimensional equivalent of matrices) and (tensor) products.

Original: We relay below the core notation used in our work, the interested reader can ﬁnd further information in the tensor-related literature (Kolda & Bader, 2009; Debals & De Lathauwer, 2017).
Revised: We relay below the core notation used in our work, the interested reader can ﬁnd further information in the tensor-related literature (Kolda & Bader, 2009; Debals & De Lathauwer, 2017).

Original: Symbols of variables : Tensors/matrices/vectors are symbolized by calligraphic/uppercase/lowercase boldface letters e.g., W , W , w .
Revised: Symbols of variables : Tensors/matrices/vectors are symbolized by calligraphic/uppercase/lowercase boldface letters e.g., W , W , w .

Original: Matrix products : The Hadamard product of A , B P R
Revised: Matrix products : The Hadamard product of A , B P R

Original: I ˆ N is deﬁned as A ˚ B and is equal to a p i,j q b p i,j q for the p
Revised: I ˆ N is deﬁned as A ˚ B and is equal to a p i,j q b p i,j q for the p

Original: i, j q element.
Revised: i, j q element.

Original: The Khatri-Rao product of matrices A P R
Revised: The Khatri-Rao product of matrices A P R

Original: I ˆ N and B P R J ˆ N is denoted by A d B and yields a matrix of dimensions p
Revised: I ˆ N and B P R J ˆ N is denoted by A d B and yields a matrix of dimensions p

Original: IJ q ˆ N .
Revised: IJ q ˆ N .

Original: The Khatri-Rao product for a set of matrices t A r m s P R
Revised: The Khatri-Rao product for a set of matrices t A r m s P R

Original: I m ˆ N u Mm “ 1 is abbreviated by A r 1 s d A r 2 s d ¨ ¨ ¨ d A r M s .
Revised: I m ˆ N u Mm “ 1 is abbreviated by A r 1 s d A r 2 s d ¨ ¨ ¨ d A r M s .

Original: “ Ä Mm “ 1 A r m s .
Revised: “ Ä Mm “ 1 A r m s .

Original: Tensors : Each element of an M th order tensor W is addressed by M indices, i.e., p W q i 1 ,i 2 ,...,i M .
Revised: Tensors : Each element of an M th order tensor W is addressed by M indices, i.e., p W q i 1 ,i 2 ,...,i M .

Original: “ w i 1 ,i 2 ,...,i M .
Revised: “ w i 1 ,i 2 ,...,i M .

Original: An M th -order tensor W is deﬁned over the tensor space R
Revised: An M th -order tensor W is deﬁned over the tensor space R

Original: I 

Revised: I

Original: ˆ I 2 ˆ¨¨¨ˆ I M , where I m P Z for m “ 1 , 2 , . . .
Revised: ˆ I 2 ˆ¨¨¨ˆ I M , where I m P Z for m “ 1 , 2 , . . .

Original: , M .
Revised: , M .

Original: The mode- m unfolding of a tensor W P R
Revised: The mode- m unfolding of a tensor W P R

Original: I 

Revised: I

Original: ˆ I 2 ˆ¨¨¨ˆ
Revised: ˆ I 2 ˆ¨¨¨ˆ

Original: I M maps W to a matrix
Revised: I M maps W to a matrix

Original: W p m q P R
Revised: W p m q P R

Original: I m ˆ
Revised: I m ˆ

Original: ¯
Revised: ¯

Original: I m with ¯ I m “ ś M k “ 1 k ‰ m
Revised: I m with ¯ I m “ ś M k “ 1 k ‰ m

Original: I k such that the tensor element w i 1 ,i 2 ,...,i M is mapped to the matrix element w i m ,j where j “ 1 ` [Equation] M k [Equation]
Revised: I k such that the tensor element w i 1 ,i 2 ,...,i M is mapped to the matrix element w i m ,j where j “ 1 ` ř M k “ 1 k ‰ m p

Original: m

Original: [Equation] k with J [Equation] “ ś 
Revised: i k ´ 1 q J k with J k “ ś k ´ 1 n “ 1 n ‰ m I n .

Original: [Equation] m I n . The mode- m vector product of W with a vector u P R I m , denoted by W ˆ m u P R
Revised: The mode- m vector product of W with a vector u P R I m , denoted by W ˆ m u P R

Original: I 

Revised: I

Original: ˆ I 2 ˆ¨¨¨ˆ I m ´
Revised: ˆ I 2 ˆ¨¨¨ˆ I m ´

Original: 1 ˆ I m ` 1 ˆ¨¨¨ˆ I M , results in a tensor of order M ´ 1 : 
Revised: 1 ˆ I m ` 1 ˆ¨¨¨ˆ I M , results in a tensor of order M ´ 1 : p

Original: [Equation]
Revised: [Equation]

Original: We denote W 
Revised: We denote W ˆ 1 u

Original: [Equation] 
Revised: [Equation] [Equation]

Revised: ˆ

Original:  The CP decomposition (Kolda & Bader, 2009) factorizes a tensor into a sum of component rank-one tensors.
Revised: [Equation] ˆ [Equation] The CP decomposition (Kolda & Bader, 2009) factorizes a tensor into a sum of component rank-one tensors.

Original: The rank- R CP decomposition of an M th -order tensor W is written as:
Revised: The rank- R CP decomposition of an M th -order tensor W is written as:

Original: [Equation]
Revised: [Equation]

Original: where ˝ is the vector outer product.
Revised: where ˝ is the vector outer product.

Original: The factor matrices
Revised: The factor matrices

Original: [Equation]
Revised: [Equation]

Original: R [Equation] ˆ R ( M m “ 1 collect the vectors from the rank-one components.
Revised: R I m ˆ R ( M m “ 1 collect the vectors from the rank-one components.

Original: By considering the mode- 1 unfolding of W , the CP decomposition can be written in matrix form as:
Revised: By considering the mode- 1 unfolding of W , the CP decomposition can be written in matrix form as:

Original: [Equation]
Revised: [Equation]

Original: The following lemma is useful in our method:
Revised: The following lemma is useful in our method:

Original: Lemma 1.
Revised: Lemma 1.

Original: For a set of N matrices t A r
Revised: For a set of N matrices t A r

Original: ν s P
Revised: ν s P

Original: R
Revised: R

Original: I ν
Revised: I ν

Original: ˆ K u Nν “ 1 and t B r
Revised: ˆ K u Nν “ 1 and t B r

Original: ν
Revised: ν

Original: s P R I ν
Revised: s P R I ν

Original: ˆ L u Nν “ 1 , the following equality holds:
Revised: ˆ L u Nν “ 1 , the following equality holds:

Original: [Equation]
Revised: [Equation]

Original: An indicative proof can be found in the Appendix of Chrysos et al.
Revised: An indicative proof can be found in the Appendix of Chrysos et al.

Original: B. S CALAR OUTPUT
Revised: B. S CALAR OUTPUT

Original: The proposed formulation expresses higher-order interactions of the input variables.
Revised: The proposed formulation expresses higher-order interactions of the input variables.

Original: To elaborate that, we develop the single output case below.
Revised: To elaborate that, we develop the single output case below.

Original: That is, we focus on an element τ of the output vector, e.g., a single pixel.
Revised: That is, we focus on an element τ of the output vector, e.g., a single pixel.

Original: In the next few paragraphs, we consider the case of a scalar output x τ , with τ P r 1 , o s when the input variables are z
Revised: In the next few paragraphs, we consider the case of a scalar output x τ , with τ P r 1 , o s when the input variables are z

Original: I , z II P K d .
Revised: I , z II P K d .

Original: To avoid cluttering the notation we only refer to the scalar output with x τ in the next few paragraphs.
Revised: To avoid cluttering the notation we only refer to the scalar output with x τ in the next few paragraphs.

Original: As a reminder, the polynomial of expansion order N P N with output x P R o has the form:
Revised: As a reminder, the polynomial of expansion order N P N with output x P R o has the form:

Original: [Equation]
Revised: [Equation]

Original: We assume a second order expansion ( N “ 2 ) and let τ denote an arbitrary scalar output of x .
Revised: We assume a second order expansion ( N “ 2 ) and let τ denote an arbitrary scalar output of x .

Original: The ﬁrst order correlations can be expressed through the sums ř dλ “ 1 w r 1 , 1 s τ,λ z II ,λ and ř dλ “ 1 w r 1 , 2 s τ,λ z I ,λ .The + [Figure]
Revised: The ﬁrst order correlations can be expressed through the sums ř dλ “ 1 w r 1 , 1 s τ,λ z II ,λ and ř dλ “ 1 w r 1 , 2 s τ,λ z I ,λ .

Original: [Figure]

Original: [Figure]

Original: [Figure] II [Figure] [Figure]
Revised: The second order correlations include both auto- and cross-correlations.

Original: [Figure]

Original: z [Figure] λ ... (cid:2) + [Figure]
Revised: The tensors W r 2 , 1 s and W r 2 , 3 s capture the auto-correlations, while the tensor W r 2 , 2 s captures the cross-correlations.

Original: +

Original: [Figure]
Revised: [Figure]

Original: second order correlations include both auto- and cross-correlations.

Original: The tensors W r 2 , 1 s and W r 2 , 3 s capture the auto-correlations, while the tensor W r 2 , 2 s captures the cross-correlations.

Original: A pictorial representation of the correlations are captured in Fig.
Revised: A pictorial representation of the correlations are captured in Fig.

Original: Collecting all the terms in an equation, each output is expressed as:
Revised: Collecting all the terms in an equation, each output is expressed as:

Original: [Equation]
Revised: [Equation]

Original: I ,λ ` d ÿ µ “w r 2 , 1 s τ,λ,µ z II ,λ z II ,µ ` d ÿ µ “w r 2 , 3 s τ,λ,µ z

Original: I ,λ

Original: z I ,µ ` d ÿ µ “w r 2 , 2 s τ,λ,µ z
Revised: z I ,µ ` d ÿ µ “w r 2 , 2 s τ,λ,µ z

Original: I ,λ z II ,µ ı (8) where β τ P R .
Revised: I ,λ z II ,µ ı (8) where β τ P R .

Original: Notice that all the correlations of up to second order are captured in equation 8.
Revised: Notice that all the correlations of up to second order are captured in equation 8.

Original: B. S ECOND ORDER DERIVATION FOR TWO - VARIABLE INPUT
Revised: B. S ECOND ORDER DERIVATION FOR TWO - VARIABLE INPUT

Original: In all our derivations, the variables associated with the ﬁrst input z I have an I notation, e.g., U r 1 ,I s .
Revised: In all our derivations, the variables associated with the ﬁrst input z I have an I notation, e.g., U r 1 ,I s .

Original: Respectively for the second input z II , the notation II is used.
Revised: Respectively for the second input z II , the notation II is used.

Original: Even though equation 7 enables any order of expansion, the learnable parameters increase exponentially, therefore we can use a coupled factorization to reduce the parameters.
Revised: Even though equation 7 enables any order of expansion, the learnable parameters increase exponentially, therefore we can use a coupled factorization to reduce the parameters.

Original: Next, we derive the factorization for a second order expansion (i.e., N “ 2 ) and then provide the recursive relationship that generalizes it for an arbitrary order.
Revised: Next, we derive the factorization for a second order expansion (i.e., N “ 2 ) and then provide the recursive relationship that generalizes it for an arbitrary order.

Original: Second order derivation : For a second order expansion (i.e., N “ 2 in equation 1), we factorize each parameter tensor W r n,ρ s .
Revised: Second order derivation : For a second order expansion (i.e., N “ 2 in equation 1), we factorize each parameter tensor W r n,ρ s .

Original: We assume a coupled CP decomposition for each parameter as follows:
Revised: We assume a coupled CP decomposition for each parameter as follows:

Original: • Let W r 1 , 1 s p 1 q “ CU T r 1 ,II s and W r 1 , 2 s p 1 q “ CU T r 1 ,I s be the parameters for n “ 1 . •
Revised: • Let W r 1 , 1 s p 1 q “ CU T r 1 ,II s and W r 1 , 2 s p 1 q “ CU T r 1 ,I s be the parameters for n “ 1 . •

Original: Let W r 2 , 1 s p 1 q “ C p U r 2 ,II s d U r 1 ,II s q T and W r 2 , 3 s p 1 q “ C p U r 2 ,I s d U r 1 ,I s q T capture the second order correlations of a single variable ( z II and z I respectively).
Revised: Let W r 2 , 1 s p 1 q “ C p U r 2 ,II s d U r 1 ,II s q T and W r 2 , 3 s p 1 q “ C p U r 2 ,I s d U r 1 ,I s q T capture the second order correlations of a single variable ( z II and z I respectively).

Original: • The cross-terms are expressed in W r 2 , 2 s ˆ 

Revised: • The cross-terms are expressed in W r 2 , 2 s ˆ

Original: z
Revised: z

Original: I ˆ 3 z II .
Revised: I ˆ 3 z II .

Original: The output of the τ element 2 is ř dλ,µ “ 1 w r 2 , 2 s τ,λ,µ z
Revised: The output of the τ element 2 is ř dλ,µ “ 1 w r 2 , 2 s τ,λ,µ z

Original: I ,λ z II ,µ .
Revised: I ,λ z II ,µ .

Original: The product ˆ W r 2 , 2 s ˆ 2 z II {+ˆ 
+}
Revised: The product ˆ W r 2 , 2 s ˆ 2 z II {+ˆ 
+}

Original: ˆ 3 z I also results in the same elementwise expression.
Revised: z I also results in the same elementwise expression.

Original: Hence, to allow for symmetric expression, we factorize the term W r 2 , 2 s p 1 q as the sum of the two terms C p U r 2 ,II s d U r 1 ,I s q T and C p
Revised: Hence, to allow for symmetric expression, we factorize the term W r 2 , 2 s p 1 q as the sum of the two terms C p U r 2 ,II s d U r 1 ,I s q T and C p

Original: U r 2 ,I s d U r 1 ,II s q T .
Revised: U r 2 ,I s d U r 1 ,II s q T .

Original: For each of the two terms, we assume that the vector-valued inputs are accordingly multiplied.
Revised: For each of the two terms, we assume that the vector-valued inputs are accordingly multiplied.

Original: The parameters C P R
Revised: The parameters C P R

Original: o ˆ k , U r m,φ s P R d
Revised: o ˆ k , U r m,φ s P R d

Original: ˆ k
Revised: ˆ k

Original: ( m “ 1 , 2 and φ “ t I, II u ) are learnable.
Revised: ( m “ 1 , 2 and φ “ t I, II u ) are learnable.

Original: The aforementioned factorization results in the following equation:
Revised: The aforementioned factorization results in the following equation:

Original: [Equation]
Revised: [Equation]

Original: This expansion captures the correlations (up to second order) of the two input variables
Revised: This expansion captures the correlations (up to second order) of the two input variables

Original: z
Revised: z

Original: I , z II .
Revised: I , z II .

Original: To make the proof more complete, we remind the reader that the recursive relationship (i.e., (2) in the main paper) is:
Revised: To make the proof more complete, we remind the reader that the recursive relationship (i.e., (2) in the main paper) is:

Original: [Equation]
Revised: [Equation]

Original: for n “ 2 , . . .
Revised: for n “ 2 , . . .

Original: , N with x 1 “ U T r 1 ,I s z
Revised: , N with x 1 “ U T r 1 ,I s z

Original: I ` U T r 1 ,II s z II and x “ Cx N ` β .
Revised: I ` U T r 1 ,II s z II and x “ Cx N ` β .

Original: Claim 1.
Revised: Claim 1.

Original: The equation (9) is a special format of a polynomial that is visualized as in Fig.
Revised: The equation (9) is a special format of a polynomial that is visualized as in Fig.

Original: 1 of the main paper.
Revised: 1 of the main paper.

Original: Equivalently, prove that (9) follows the recursive relationship of (10).
Revised: Equivalently, prove that (9) follows the recursive relationship of (10).

Original: Proof.
Revised: Proof.

Original: We observe that the ﬁrst two terms of equation 9 are equal to Cx 1 (from equation 10).
Revised: We observe that the ﬁrst two terms of equation 9 are equal to Cx 1 (from equation 10).

Original: By applying Lemma 1 in the terms that have Khatri-Rao product, we obtain:
Revised: By applying Lemma 1 in the terms that have Khatri-Rao product, we obtain:

Original: [Equation]
Revised: [Equation]

Original: The last equation is precisely the one that arises from the recursive relationship from equation 10.
Revised: The last equation is precisely the one that arises from the recursive relationship from equation 10.

Original: To prove the recursive formula for the N th order expansion, a similar pattern as in sec.
Revised: To prove the recursive formula for the N th order expansion, a similar pattern as in sec.

Original: C of PolyGAN (Chrysos et al., 2019) can be followed.
Revised: C of PolyGAN (Chrysos et al., 2019) can be followed.

Original: Speciﬁcally, the difference here is that because of the two input variables, the auto- and cross-correlation variables should be included.
Revised: Speciﬁcally, the difference here is that because of the two input variables, the auto- and cross-correlation variables should be included.

Original: Other than that, the same factor sharing is followed.
Revised: Other than that, the same factor sharing is followed.

Original: B. N ESTED -MVP MODEL FOR TWO - VARIABLE INPUT
Revised: B. N ESTED -MVP MODEL FOR TWO - VARIABLE INPUT

Original: The model proposed above (i.e., equation 10), relies on a single coupled CP decomposition, however a more ﬂexible model can factorize each level with a CP decomposition.
Revised: The model proposed above (i.e., equation 10), relies on a single coupled CP decomposition, however a more ﬂexible model can factorize each level with a CP decomposition.

Original: To effectively do that, we utilize learnable hyper-parameters b r n s P R ω for n P r 1 , N s , which act as scaling factors for each parameter tensor.
Revised: To effectively do that, we utilize learnable hyper-parameters b r n s P R ω for n P r 1 , N s , which act as scaling factors for each parameter tensor.

Original: Then, a polynomial of expansion order N P N with output x P R o has the form:
Revised: Then, a polynomial of expansion order N P N with output x P R o has the form:

Original: [Equation]
Revised: [Equation]

Original: To demonstrate the factorization without cluttering the notation, we assume a second order expansion in equation 12.
Revised: To demonstrate the factorization without cluttering the notation, we assume a second order expansion in equation 12.

Original: Second order derivation : The second order expansion, i.e., N “ 2 , is derived below.
Revised: Second order derivation : The second order expansion, i.e., N “ 2 , is derived below.

Original: We jointy factorize all parameters of equation 12 with a nested decomposition as follows:
Revised: We jointy factorize all parameters of equation 12 with a nested decomposition as follows:

Original: • First order parameters : W r 1 , 1 s p 1 q “ C p
Revised: • First order parameters : W r 1 , 1 s p 1 q “ C p

Original: A r 2 ,II s d B r 2 s q T and W r 1 , 2 s p 1 q
Revised: A r 2 ,II s d B r 2 s q T and W r 1 , 2 s p 1 q

Original: [Equation]
Revised: [Equation]

Original: • Let W r 2 ,
Revised: p

Original: [Equation]
Revised: [Equation]

Original:  capture the second order correlations of a single variable ( z II and z I respectively).
Revised: 2 ,I s d B r 2 s q T . • Let W r 2 , 1 s p 1 q “ C " A r 2 ,II s d „´ A r 1 ,II s d B r 1 s ¯ V r 2 s * T and W r 2 , 3 s p 1 q “ C " A r 2 ,I s d „´ A r 1 ,I s d B r 1 s ¯ V r 2 s * T capture the second order correlations of a single variable ( z II and z I respectively).

Original: • The cross-terms are included in W r 2 , 2 s ˆ 2 b r 1 s ˆ 

Revised: • The cross-terms are included in W r 2 , 2 s ˆ 2 b r 1 s ˆ

Original: z
Revised: z

Original: I ˆ 4 z II .
Revised: I ˆ 4 z II .

Original: The output of the τ element is expressed as ř ων “ 1 ř dλ,µ “ 1 w r 2 , 2 s τ,ν,λ,µ b r 1 s ,ω
Revised: The output of the τ element is expressed as ř ων “ 1 ř dλ,µ “ 1 w r 2 , 2 s τ,ν,λ,µ b r 1 s ,ω

Original: z I ,λ z II ,µ .
Revised: z I ,λ z II ,µ .

Original: Similarly, the product ˆ W r 2 , 2 s ˆ 2 b r 1 s ˆz II
Revised: Similarly, the product ˆ W r 2 , 2 s ˆ 2 b r 1 s ˆ z

Original: [-ˆ 
-]
Revised: [-ˆ 
-][Equation]

Original: z I has output
Revised: z I has output

Original: [Equation]
Revised: [Equation]

Original: ω z I ,λ z II ,µ for the τ element.
Revised: ,λ z II ,µ for the τ element.

Original: Notice that the only change in the two expressions is the permutation of the third and forth modes of the tensor; the rest of the expression remains the same.
Revised: Notice that the only change in the two expressions is the permutation of the third and forth modes of the tensor; the rest of the expression remains the same.

Original: Therefore, to account for this symmetry we factorize the term W r 2 , 2 s as the sum of two terms and assume that each term is multiplied by the respective terms.
Revised: Therefore, to account for this symmetry we factorize the term W r 2 , 2 s as the sum of two terms and assume that each term is multiplied by the respective terms.

Original: Let W r 2 , 2 s p 1 q
Revised: Let W r 2 , 2 s p 1 q

Original: [Equation]
Revised: [Equation]

Original: The parameters C P R o ˆ k , A r n,φ s P R d ˆ k , V r n s P R k ˆ k , B r n s P R ω ˆ k for n “ 1 , 2 and φ “ t I, II u are learnable.
Revised: The parameters C P R o ˆ k , A r n,φ s P R d ˆ k , V r n s P R k ˆ k , B r n s P R ω ˆ k for n “ 1 , 2 and φ “ t I, II u are learnable.

Original: Collecting all the terms above and extracting C as a common factor (we ommit
Revised: Collecting all the terms above and extracting C as a common factor (we ommit

Original: C below to avoid cluttering the notation):
Revised: C below to avoid cluttering the notation):

Original: [Equation]
Revised: [Equation]

Revised: A r 2 ,II s d „´

Revised: A r 1 ,I s d B

Revised: r 1 s ¯

Revised: V r 2 s * T p z II d z I d b r 1 s q “ ´

Revised: A T r 2 ,II s z II ` A T r 2 ,I s z

Revised: I ¯ ˚ ´

Revised: B T r 2 s b r 2 s ¯ ` ´

Revised: A T r 2 ,II s z II ` A T r 2 ,I s z

Revised: I ¯ ˚ "

Revised: V T r 2 s „´

Revised: A T r 1 ,II s z II ` A T r 1 ,I s z

Revised: I ¯ ˚ ´

Revised: B T r 1 s b r 1 s ¯* (13)

Original: The last equation is precisely a recursive equation that can be expressed with the Fig.
Revised: The last equation is precisely a recursive equation that can be expressed with the Fig.

Original: 5 or equivalently the generalized recursive relationship below. 
Revised: 5 or equivalently the generalized recursive relationship below. [Equation]

Original: Recursive relationship : The recursive formula for the Nested-MVP model with arbitrary expansion order N P N is the following:
Revised: Recursive relationship : The recursive formula for the Nested-MVP model with arbitrary expansion order N P N is the following:

Original: [Equation]
Revised: [Equation]

Original: 14) where n P r 2 , N s and x 1 “ ´
Revised: where n P r 2 , N s and x 1 “ ´

Original: A T r 1 ,I s z
Revised: A T r 1 ,I s z

Original: I ` A T r 1 ,II s z II ¯ ˚ ´
Revised: I ` A T r 1 ,II s z II ¯ ˚ ´

Original: B T r 1 s b r 1 s ¯ .
Revised: B T r 1 s b r 1 s ¯ .

Original: The parameters C P
Revised: The parameters C P

Original: R o ˆ k , A r n,φ s P R d ˆ k , V r n s P R k ˆ k , B r n s P R ω
Revised: R o ˆ k , A r n,φ s P R d ˆ k , V r n s P R k ˆ k , B r n s P R ω

Original: ˆ k for φ “ t I, II u are learnable.
Revised: ˆ k for φ “ t I, II u are learnable.

Original: Then, the output x “ Cx N ` β .
Revised: Then, the output x “ Cx N ` β .

Original: The Nested-MVP model manifests an alternative network that relies on slightly modiﬁed assumptions on the decomposition.
Revised: The Nested-MVP model manifests an alternative network that relies on slightly modiﬁed assumptions on the decomposition.

Original: Thus, changing the underlying assumptions of the decomposition can modify the resulting network.
Revised: Thus, changing the underlying assumptions of the decomposition can modify the resulting network.

Original: This can be an important tool for domain-speciﬁc applications, e.g., when the domain-knowledge should be inserted in the last layers.
Revised: This can be an important tool for domain-speciﬁc applications, e.g., when the domain-knowledge should be inserted in the last layers.

Original: 𝑧 𝐼 𝑧 𝐼𝐼 + 𝐴 1,𝐼𝐼 𝐴 1,𝐼 ∗ + 𝐴 2,𝐼𝐼 𝐴 2,𝐼 + … ∗ + 𝐴 𝑁,𝐼𝐼 𝐴 𝑁,𝐼 𝐶 + 𝛽 𝐺(𝑧 𝐼 , 𝑧 𝐼𝐼 ) ∼ ∼ 𝐵𝑀𝑊,… or 𝑉𝐵 [2] 𝑏 [2] + 𝑉 𝑁 𝐵

Original: [𝑁] 𝑏

Original: [𝑁]

Original: C B EYOND TWO VARIABLES
Revised: C B EYOND TWO VARIABLES

Original: Frequently, more than one conditional inputs are required (Yu et al., 2018b; Xu et al., 2017; Maximov et al., 2020).
Revised: Frequently, more than one conditional inputs are required (Yu et al., 2018b; Xu et al., 2017; Maximov et al., 2020).

Original: In such tasks, the aforementioned framework can be generalized to more than two input variables.
Revised: In such tasks, the aforementioned framework can be generalized to more than two input variables.

Original: We demonstrate how this is possible with three variables; then it can trivially extended to an arbitrary number of input variables.
Revised: We demonstrate how this is possible with three variables; then it can trivially extended to an arbitrary number of input variables.

Original: Let z I , z II , z III P K d denote the three input variables.
Revised: Let z I , z II , z III P K d denote the three input variables.

Original: We aim to learn a function that captures the higher-order interactions of the input variables.
Revised: We aim to learn a function that captures the higher-order interactions of the input variables.

Original: The polynomial of expansion order N P N with output x P R o has the form:
Revised: The polynomial of expansion order N P N with output x P R o has the form:

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: where β P R o and W r n,ρ,δ s P R o ˆ ś nm
Revised: where β P R o and W r n,ρ,δ s P R o ˆ ś nm

Original: “ 1 ˆ m d (for n P r 1 , N s and ρ, δ P r 1 , n ` 1 s ) are the learnable parameters.
Revised: “ 1 ˆ m d (for n P r 1 , N s and ρ, δ P r 1 , n ` 1 s ) are the learnable parameters.

Original: As in the two-variable input, the unknown parameters increase exponentially.
Revised: As in the two-variable input, the unknown parameters increase exponentially.

Original: To that end, we utilize a joint factorization with factor sharing.
Revised: To that end, we utilize a joint factorization with factor sharing.

Original: The recursive relationship of such a factorization is:
Revised: The recursive relationship of such a factorization is:

Original: [Equation]
Revised: [Equation]

Original: for n “ 2 , . . .
Revised: for n “ 2 , . . .

Original: , N with x 1 “ U T r 1 ,I s z
Revised: , N with x 1 “ U T r 1 ,I s z

Original: I ` U T r 1 ,II s z II ` U T r 1 ,III s z III and x “ Cx N ` β .
Revised: I ` U T r 1 ,II s z II ` U T r 1 ,III s z III and x “ Cx N ` β .

Original: Notice that the pattern (for each order) is similar to the two-variable input: a) a different embedding is found for each input variable, b) the embeddings are added together, c) the result is multiplied elementwise with the representation of the previous order.
Revised: Notice that the pattern (for each order) is similar to the two-variable input: a) a different embedding is found for each input variable, b) the embeddings are added together, c) the result is multiplied elementwise with the representation of the previous order.

Original: D C ONCATENATION OF INPUTS
Revised: D C ONCATENATION OF INPUTS

Original: A popular method used for conditional generation is to concatenate the conditional input with the noise labels.
Revised: A popular method used for conditional generation is to concatenate the conditional input with the noise labels.

Original: However, as we showcase below, concatenation has two signiﬁcant drawbacks when compared to our framework.
Revised: However, as we showcase below, concatenation has two signiﬁcant drawbacks when compared to our framework.

Original: To explain those, we will deﬁne a concatenation model.
Revised: To explain those, we will deﬁne a concatenation model.

Original: Let z
Revised: Let z

Original: I P K d 1 1 , z II P K d 2 2 where K 1 , K 2 can be a subset of real or natural numbers.
Revised: I P K d 1 1 , z II P K d 2 2 where K 1 , K 2 can be a subset of real or natural numbers.

Original: The output of a concatenation layer is x “ P T ” z
Revised: The output of a concatenation layer is x “ P T ” z

Original: I ; z II ı T where the symbol ‘ ; ’ denotes the concatenation and P P R p d 1 d 2 qˆ o is an afﬁne transformation on the concatenated vector.
Revised: I ; z II ı T where the symbol ‘ ; ’ denotes the concatenation and P P R p d 1 d 2 qˆ o is an afﬁne transformation on the concatenated vector.

Original: The j th output is 
Revised: The j th output is x

Original: [Equation]
Revised: [Equation]

Original: Therefore, the two differences from the concatenation case are:
Revised: Therefore, the two differences from the concatenation case are:

Original: • If the input variables are concatenated together we obtain an additive format, not a multiplicative that can capture cross-term correlations.
Revised: • If the input variables are concatenated together we obtain an additive format, not a multiplicative that can capture cross-term correlations.

Original: That is, the multiplicative format does allow achieving higher-order auto- and cross- term correlations.
Revised: That is, the multiplicative format does allow achieving higher-order auto- and cross- term correlations.

Original: • The concatenation changes the dimensionality of the embedding space.
Revised: • The concatenation changes the dimensionality of the embedding space.

Original: Speciﬁcally, the input space has dimensionality d 1 ¨ d 2 .
Revised: Speciﬁcally , the input space has dimensionality d 1 ¨ d 2 .

Original: That has a signiﬁcant toll on the size of the ﬁlters (i.e., it increases the learnable parameters), while still having an additive impact.
Revised: That has a signiﬁcant toll on the size of the ﬁlters (i.e., it increases the learnable parameters), while still having an additive impact.

Original: On the contrary, our framework does not change the dimensionality of the embedding spaces.
Revised: On the contrary, our framework does not change the dimensionality of the embedding spaces.

Original: E I N - DEPTH DIFFERENCES FROM Π -N ET
Revised: E I N - DEPTH DIFFERENCES FROM Π -N ET

Original: In the next few paragraphs, we conduct an in-depth analysis of the differences between Π -Net and
Revised: In the next few paragraphs, we conduct an in-depth analysis of the differences between Π -Net and

Original: MVP.
Revised: MVP.

Original: The analysis assumes knowledge of the proposed model, i.e., (2).
Revised: The analysis assumes knowledge of the proposed model, i.e., (2).

Original: Chrysos et al.
Revised: Chrysos et al.

Original: (2020) introduce Π -Net as a polynomial expansion of a single input variable.
Revised: (2020) introduce Π -Net as a polynomial expansion of a single input variable.

Original: Their goal is to model functions x “ G p z q as high-order polynomial expansions of z .
Revised: Their goal is to model functions x “ G p z q as high-order polynomial expansions of z .

Original: Their focus is towards using a single-input variable z , which can be noise in case of image generation or an image in discriminative experiments.
Revised: Their focus is towards using a single-input variable z , which can be noise in case of image generation or an image in discriminative experiments.

Original: The authors express the StyleGAN architecture (Karras et al., 2019) as a polynomial expansion, while they advocate that the impressive results can be attributed to the polynomial expansion.
Revised: The authors express the StyleGAN architecture (Karras et al., 2019) as a polynomial expansion, while they advocate that the impressive results can be attributed to the polynomial expansion.

Original: To facilitate the in-depth analysis, the recursive relationship that corresponds to (2) is provided below.
Revised: To facilitate the in-depth analysis, the recursive relationship that corresponds to (2) is provided below.

Original: An N th order expansion in Π -Net is expressed as:
Revised: An N th order expansion in Π -Net is expressed as:

Original: [Equation]
Revised: [Equation]

Original: (17) for n “ 2 , . . .
Revised: (17) for n “ 2 , . . .

Original: , N with x 1 “ Λ T r 1 s z and x “ Γ x N ` β .
Revised: , N with x 1 “ Λ T r 1 s z and x “ Γ x N ` β .

Original: The parameters Λ , Γ are learnable.
Revised: The parameters Λ , Γ are learnable.

Original: In this work, we focus on conditional data generation, i.e., there are multiple input variables available as auxiliary information.
Revised: In this work, we focus on conditional data generation, i.e., there are multiple input variables available as auxiliary information.

Original: The trivial application of Π -Net would be to concatenate all the M input variables
Revised: The trivial application of Π -Net would be to concatenate all the M input variables

Original: z I , z II , z III , . . . .
Revised: z I , z II , z III , . . . .

Original: The input variable z becomes z “ ” z I ; z II ; z III ; . . .
Revised: The input variable z becomes z “ ” z I ; z II ; z III ; . . .

Original: ı , where the symbol ‘ ; ’ denotes the concatenation.
Revised: ı , where the symbol ‘ ; ’ denotes the concatenation.

Original: Then, the polynomial expansion of Π -Net can be learned on the concatenated z .
Revised: Then, the polynomial expansion of Π -Net can be learned on the concatenated z .

Original: However, there are four signiﬁcant reasons that we believe that this is not as ﬂexible as the proposed MVP.
Revised: However, there are four signiﬁcant reasons that we believe that this is not as ﬂexible as the proposed MVP.

Original: When we refer to Π -Net below, we refer to the model with concatenated input.
Revised: When we refer to Π -Net below, we refer to the model with concatenated input.

Original: In addition, let z I P K d 1 1 , z II P K d 2 2 denote the input variables where K 1 , K 2 can be a subset of real or natural numbers.
Revised: In addition, let z I P K d 1 1 , z II P K d 2 2 denote the input variables where K 1 , K 2 can be a subset of real or natural numbers.

Original: Parameter sharing: MVP allows additional ﬂexibility in the structure of the architecture, since
Revised: Parameter sharing: MVP allows additional ﬂexibility in the structure of the architecture, since

Original: MVP utilizes a different projection layer for each input variable.
Revised: MVP utilizes a different projection layer for each input variable.

Original: We utilize this ﬂexibility to share the parameters of the conditional input variable; as we detail in (19), we set U r n,II s “ U r 1 ,II s on (2).
Revised: We utilize this ﬂexibility to share the parameters of the conditional input variable; as we detail in (19), we set U r n,II s “ U r 1 ,II s on (2).

Original: If we want to perform a similar sharing in Π -Net, the formulation equivalent to (17) would be p λ r n s q i “ p λ r 1 s q i for i “ d 1 , . . .
Revised: If we want to perform a similar sharing in Π -Net, the formulation equivalent to (17) would be p λ r n s q i “ p λ r 1 s q i for i “ d 1 , . . .

Original: , d 1 ` d 2 .
Revised: , d 1 ` d 2 .

Original: However, sharing only part of the matrix might be challenging.
Revised: However, sharing only part of the matrix might be challenging.

Original: Additionally, when Λ is a convolution, the sharing pattern is not straightforward to be computed.
Revised: Additionally, when Λ is a convolution, the sharing pattern is not straightforward to be computed.

Original: Therefore, MVP enables additional ﬂexibility to the model, which is hard to be included in Π -Net.
Revised: Therefore, MVP enables additional ﬂexibility to the model, which is hard to be included in Π -Net.

Original: Inductive bias: The inductive bias is crucial in machine learning (Zhao et al., 2018), however concatenating the variables restricts the ﬂexibility of the model (i.e. Π -Net).
Revised: Inductive bias: The inductive bias is crucial in machine learning (Zhao et al., 2018), however concatenating the variables restricts the ﬂexibility of the model (i.e. Π -Net).

Original: To illustrate that, let us use the super-resolution experiments as an example.
Revised: To illustrate that, let us use the super-resolution experiments as an example.

Original: The input variable z I is the noise vector and z II is the (vectorized) low-resolution image.
Revised: The input variable z I is the noise vector and z II is the (vectorized) low-resolution image.

Original: If we concatenate the two variables, then we should use a fully-connected (dense) layer, which does not model well the spatial correlations.
Revised: If we concatenate the two variables, then we should use a fully-connected (dense) layer, which does not model well the spatial correlations.

Original: Instead, with MVP, we use a fully-connected layer for the noise vector and a convolution for z II (low-resolution image).
Revised: Instead, with MVP, we use a fully-connected layer for the noise vector and a convolution for z II (low-resolution image).

Original: The convolution reduces the number of parameters and captures the spatial correlations in the image.
Revised: The convolution reduces the number of parameters and captures the spatial correlations in the image.

Original: Thus, by concatenating the variables, we reduce the ﬂexibility of the model.
Revised: Thus, by concatenating the variables, we reduce the ﬂexibility of the model.

Original: Dimensionality of the inputs: The dimensionality of the inputs might vary orders of magnitude, which might create an imbalance during learning.
Revised: Dimensionality of the inputs: The dimensionality of the inputs might vary orders of magnitude, which might create an imbalance during learning.

Original: For instance, in class-conditional generation concatenating the one-hot labels in the input does not scale well when there are hundreds of classes (Odena et al., 2017).
Revised: For instance, in class-conditional generation concatenating the one-hot labels in the input does not scale well when there are hundreds of classes (Odena et al., 2017).

Original: We observe a similar phenomenon in class-conditional generation: in Cars196 (with 196 classes)
Revised: We observe a similar phenomenon in class-conditional generation: in Cars196 (with 196 classes)

Original: the performance of Π -Net deteriorates considerably when compared to its (relative) performance in CIFAR10 (with 10 classes).
Revised: the performance of Π -Net deteriorates considerably when compared to its (relative) performance in CIFAR10 (with 10 classes).

Original: On the contrary, MVP does not fuse the elements of the input variables directly, but it projects them into a subspace appropriate for adding them.
Revised: On the contrary, MVP does not fuse the elements of the input variables directly, but it projects them into a subspace appropriate for adding them.

Original: Order of expansion with respect to each variable:
Revised: Order of expansion with respect to each variable:

Original: Frequently, the two inputs do not require the same order of expansion.
Revised: Frequently, the two inputs do not require the same order of expansion.

Original: Without loss of generality, assume that we need correlations up to N I and
Revised: Without loss of generality, assume that we need correlations up to N I and

Original: N II order (with N I ă N II ) from z
Revised: N II order (with N I ă N II ) from z

Original: I and z II respectively.
Revised: I and z II respectively.

Original: MVP includes a different transformation for each variable, i.e., U r n,I s for z
Revised: MVP includes a different transformation for each variable, i.e., U r n,I s for z

Original: I and U r n,II s for z II .
Revised: I and U r n,II s for z II .

Original: Then, we can set U r n,I s “ 0 for n ą N I .
Revised: Then, we can set U r n,I s “ 0 for n ą N I .

Original: On the contrary, the concatenation of inputs (in Π -Net) constrains the expansion to have the same order with respect to each variable.
Revised: On the contrary, the concatenation of inputs (in Π -Net) constrains the expansion to have the same order with respect to each variable.

Original: All in all, we can use concatenation to fuse variables and use Π -Net, however an inherently multivariate model is more ﬂexible and can better encode the types of inductive bias required for conditional data generation. 
Revised: All in all, we can use concatenation to fuse variables and use Π -Net, however an inherently multivariate model is more ﬂexible and can better encode the types of inductive bias required for conditional data generation. F D IFFERENCES FROM OTHER NETWORKS CAST AS POLYNOMIAL NEURAL NETWORKS

Revised: A number of networks with impressive results have emerged in (conditional) data generation the last few years.

Revised: Three such networks that are particularly interesting in our context are Karras et al.

Original: F I N - DEPTH DIFFERENCES FROM SPADE
Revised: ; Park et al. ; Chen et al. We analyze below each method and how it relates to polynomial expansions: • Karras et al.

Revised: (2019) propose an Adaptive instance normalization (AdaIN) method for unsupervised image generation.

Revised: An AdaIN layer expresses a second-order interaction 3 : h “ p Λ T w q ˚ n p c p h in qq , where n is a normalization, c the convolution operator and w is the transformed noise w “ MLP p

Revised: z I q (mapping network).

Revised: The parameters Λ are learnable, while h in is the input to the AdaIN.

Revised: Stacking AdaIN layers results in a polynomial expansion with a single variable.

Revised: • Chen et al.

Revised: (2019) propose a normalization method, called sBN, to stabilize the GAN training.

Revised: The method performs a ‘self-modulation’ with respect to the noise variable and optionally the conditional variable in the class-conditional generation setting.

Revised: Henceforth, we focus on the class-conditional setting that is closer to our work.

Revised: sBN injects the network layers with a multiplicative interaction of the input variables.

Revised: Speciﬁcally, sBN projects the conditional variable into the space of the variable z I through an embedding function.

Revised: Then, the interaction of the two vector-like variables is passed through a fully-connected layer (and a ReLU activation function); the result is injected into the network through the batch normalization parameters.

Revised: If cast as a polynomial expansion, a network with sBN layers expresses a single polynomial expansion• Park et al.

Revised: (2019) introduce a spatially-adaptive normalization, i.e., SPADE, to improve semantic image synthesis.

Revised: Their model, referred to as SPADE in the remainder of this work, assumes a semantic layout as a conditional input that facilitates the image generation.

Revised: We analyze in sec.

Revised: F.1 how to obtain the formulation of their spatially-adaptive normalization.

Revised: If cast as a polynomial expansion, SPADE expresses a polynomial expansion with respect to the conditional variable.

Revised: The aforementioned works propose or modify the batch normalization layer to improve the performance or stabilize the training, while in our work we propose the multivariate polynomial as a general function approximation technique for conditional data generation.

Revised: Nevertheless, given the interpretation of the previous works in the perspective of polynomials, we still can express them as special cases of MVP.

Revised: Methodologically, there are two signiﬁcant limitations that none of the aforementioned works tackle:

Revised: • The aforementioned architectures focus on no or one conditional variable.

Revised: Extending the frameworks to multiple conditional variables might not be trivial, while MVP naturally extends to arbitrarily many conditional variables.

Revised: • Even though the aforementioned three architectures use (implicitly) a polynomial expansion, a signiﬁcant factor is the order of the expansion.

Revised: In our work, the product of polynomials enables capturing higher-order correlations without increasing the amount of layers substantially (sec. 3.2).

Revised: In addition to the aforementioned methodological differences, our work is the only polynomial expansion that conducts experiments on a variety of conditional data generation tasks .

Revised: Thus, we both demonstrate methodologically and verify experimentally that MVP can be used for a wide range of conditional data generation tasks.

Original: In the next few paragraphs, we conduct an in-depth analysis of the differences between SPADE and
Revised: In the next few paragraphs, we conduct an in-depth analysis of the differences between SPADE and

Original: MVP.
Revised: MVP.

Original: Park et al.
Revised: Park et al.

Original: (2019) introduce a spatially-adaptive normalization, i.e., SPADE, to improve semantic image synthesis.
Revised: (2019) introduce a spatially-adaptive normalization, i.e., SPADE, to improve semantic image synthesis.

Original: Their model, referred to as SPADE in the remainder of this work, assumes a semantic layout as a conditional input that facilitates the image generation.
Revised: Their model, referred to as SPADE in the remainder of this work, assumes a semantic layout as a conditional input that facilitates the image generation.

Original: The n th model block applies a normalization on the representation x n ´ 1 of the previous layer and then it performs an elementwise multiplication with a transformed semantic layout.
Revised: The n th model block applies a normalization on the representation x n ´ 1 of the previous layer and then it performs an elementwise multiplication with a transformed semantic layout.

Original: The transformed semantic layout can be denoted as A T r n,II s z II where z II denotes the conditional input to the generator.
Revised: The transformed semantic layout can be denoted as A T r n,II s z II where z II denotes the conditional input to the generator.

Original: The output of this elementwise multiplication is then propagated to the next model block that performs the same operations.
Revised: The output of this elementwise multiplication is then propagated to the next model block that performs the same operations.

Original: Stacking N such blocks results in an N th order polynomial expansion which is expressed as:
Revised: Stacking N such blocks results in an N th order polynomial expansion which is expressed as:

Original: [Equation]
Revised: [Equation]

Original: A T r n,II s z II ¯ ˚ ´ V T r n s x n ´ 1 ` B T r n s b r n s ¯ (18) where n P r 2 , N s and x 1 “ A T r 1 ,I s z I .
Revised: (18) where n P r 2 , N s and x 1 “ A T r 1 ,I s z I .

Original: The parameters C P R o ˆ k , A r n,φ s P R d ˆ k , V r n s P
Revised: The parameters C P R o ˆ k , A r n,φ s P R d ˆ k , V r n s P

Original: R k ˆ k , B r n s P R ω ˆ k for φ “ t I, II u are learnable.
Revised: R k ˆ k , B r n s P R ω ˆ k for φ “ t I, II u are learnable.

Original: Then, the output x “ Cx N ` β .
Revised: Then, the output x “ Cx N ` β .

Original: SPADE as expressed in (18) resembles one of the proposed models of MVP (speciﬁcally (14)).
Revised: SPADE as expressed in (18) resembles one of the proposed models of MVP (speciﬁcally (14)).

Original: In particular, it expresses a polynomial with respect to the conditional variable.
Revised: In particular, it expresses a polynomial with respect to the conditional variable.

Original: The parameters A r n,I s are set as zero, which means that there are no higher-order correlations with respect to the input variable
Revised: The parameters A r n,I s are set as zero, which means that there are no higher-order correlations with respect to the input variable

Original: z I .
Revised: z I .

Original: Therefore, our work bears the following differences from Park et al.
Revised: Therefore, our work bears the following differences from Park et al.

Original: • SPADE proposes a normalization scheme that is only applied to semantic image generation.
Revised: • SPADE proposes a normalization scheme that is only applied to semantic image generation.

Original: On the contrary, our proposed MVP can be applied to any conditional data generation task, e.g., class-conditional generation or image-to-image translation.
Revised: On the contrary, our proposed MVP can be applied to any conditional data generation task, e.g., class-conditional generation or image-to-image translation.

Original: • SPADE is a special case of MVP .
Revised: • SPADE is a special case of MVP .

Original: In particular, by setting i)
Revised: In particular, by setting i)

Original: A r 1 ,II s equal to zero, ii)
Revised: A r 1 ,II s equal to zero, ii)

Original: A r n,I s in (14) equal to zero, we obtain SPADE.
Revised: A r n,I s in (14) equal to zero, we obtain SPADE.

Original: In addition, MVP allows different assumptions on the decompositions which lead to an alternative structure, such as (2).
Revised: In addition, MVP allows different assumptions on the decompositions which lead to an alternative structure, such as (2).

Original: • SPADE proposes a polynomial expansion with respect to a single variable.
Revised: • SPADE proposes a polynomial expansion with respect to a single variable.

Original: On the other hand, our model can extend to an arbitrary number of input variables to account for auxiliary labels, e.g., (16).
Revised: On the other hand, our model can extend to an arbitrary number of input variables to account for auxiliary labels, e.g., (16).

Original: • Even though SPADE models higher-order correlations of the conditional variable, it still does not leverage the higher-order correlations of the representations (e.g., as in the product of polynomials) and hence without activation functions it might not work as well as the two-variable expansion.
Revised: • Even though SPADE models higher-order correlations of the conditional variable, it still does not leverage the higher-order correlations of the representations (e.g., as in the product of polynomials) and hence without activation functions it might not work as well as the two-variable expansion.

Original: Park et al.
Revised: Park et al.

Original: (2019) exhibit impressive generation results with large-scale computing (i.e., they report results using NVIDIA DGX with 8 V100 GPUs).
Revised: (2019) exhibit impressive generation results with large-scale computing (i.e., they report results using NVIDIA DGX with 8 V100 GPUs).

Original: Our goal is not to compete in computationally heavy, large-scale experiments, but rather to illustrate the beneﬁts of the generic formulation of MVP.
Revised: Our goal is not to compete in computationally heavy, large-scale experiments, but rather to illustrate the beneﬁts of the generic formulation of MVP.

Original: SPADE is an important baseline for our work.
Revised: SPADE is an important baseline for our work.

Original: In particular, we augment SPADE in wo ways: a) by extending it to accept both continuous and discrete variables in z II and b) by adding polynomial terms with respect to the input variable
Revised: In particular, we augment SPADE in wo ways: a) by extending it to accept both continuous and discrete variables in z II and b) by adding polynomial terms with respect to the input variable

Original: z I .
Revised: z I .

Original: The latter model is referred to as SPADE-MVP (details on the next section).
Revised: The latter model is referred to as SPADE-MVP (details on the next section).

Original: G E XPERIMENTAL DETAILS
Revised: G E XPERIMENTAL DETAILS

Original: Metrics: The two most popular metrics (Lucic et al., 2018; Creswell et al., 2018) for evaluation of the synthesized images are the Inception Score (IS) (Salimans et al., 2016) and the Frechet Inception
Revised: Metrics: The two most popular metrics (Lucic et al., 2018; Creswell et al., 2018) for evaluation of the synthesized images are the Inception Score (IS) (Salimans et al., 2016) and the Frechet Inception

Original: Distance (FID) (Heusel et al., 2017).
Revised: Distance (FID) (Heusel et al., 2017).

Original: The metrics utilize the pretrained Inception network (Szegedy et al., 2015) to extract representations of the synthesized images.
Revised: The metrics utilize the pretrained Inception network (Szegedy et al., 2015) to extract representations of the synthesized images.

Original: FID assumes that the representations extracted follow a Gaussian distribution and matches the statistics (i.e., mean and variance) of the representations between real and synthesized samples.
Revised: FID assumes that the representations extracted follow a Gaussian distribution and matches the statistics (i.e., mean and variance) of the representations between real and synthesized samples.

Original: Alternative evaluation metrics have been reported as inaccurate, e.g., in Theis et al.
Revised: Alternative evaluation metrics have been reported as inaccurate, e.g., in Theis et al.

Original: (2016), thus we use the IS and FID.
Revised: (2016), thus we use the IS and FID.

Original: Following the standard practice of the literature, the IS is computed by synthesizing 5 , 000 samples, while the FID is computed using 10 , 000 samples.
Revised: Following the standard practice of the literature, the IS is computed by synthesizing 5 , 000 samples, while the FID is computed using 10 , 000 samples.

Original: The IS is used exclusively for images of natural scenes as a metric.
Revised: The IS is used exclusively for images of natural scenes as a metric.

Original: The reasoning behind that is that the Inception network has been trained on images of natural scenes.
Revised: The reasoning behind that is that the Inception network has been trained on images of natural scenes.

Original: On the contrary, the FID metric relies on the ﬁrst and second-order moments of the representations, which are considered more robust to different types of images.
Revised: On the contrary, the FID metric relies on the ﬁrst and second-order moments of the representations, which are considered more robust to different types of images.

Original: Hence, we only report IS for the CIFAR10 related experiments, while for the rest the FID is reported.
Revised: Hence, we only report IS for the CIFAR10 related experiments, while for the rest the FID is reported.

Original: Dataset details: There are ﬁve main datasets used in this work:
Revised: Dataset details: There are ﬁve main datasets used in this work:

Original: • Large-scale CelebFaces Attributes (or CelebA for short) (Liu et al., 2015) is a large-scale face attributes dataset with 202 , 000 celebrity images.
Revised: • Large-scale CelebFaces Attributes (or CelebA for short) (Liu et al., 2015) is a large-scale face attributes dataset with 202 , 000 celebrity images.

Original: We use 160 , 000 images for training our method.
Revised: We use 160 , 000 images for training our method.

Original: • Cars196 (Krause et al., 2013) is a dataset that includes different models of cars in different positions and backgrounds.
Revised: • Cars196 (Krause et al., 2013) is a dataset that includes different models of cars in different positions and backgrounds.

Original: Cars196 has 16 , 000 images, while the images have substantially more variation than CelebA faces.
Revised: Cars196 has 16 , 000 images, while the images have substantially more variation than CelebA faces.

Original: • CIFAR

Revised: • CIFAR

Original: (Krizhevsky et al., 2014) contains 60 , 000 images of natural scenes.
Revised: (Krizhevsky et al., 2014) contains 60 , 000 images of natural scenes.

Original: Each image is of resolution 32 ˆ 32 ˆ 3 and is classiﬁed in one of the 10 classes.
Revised: Each image is of resolution 32 ˆ 32 ˆ 3 and is classiﬁed in one of the 10 classes.

Original: CIFAR10 is frequently used as a benchmark for image generation.
Revised: CIFAR10 is frequently used as a benchmark for image generation.

Original: • The Street View House Numbers dataset (or SVHN for short) (Netzer et al., 2011) has 100 , 000 images of digits ( 73 , 257 of which for training).
Revised: • The Street View House Numbers dataset (or SVHN for short) (Netzer et al., 2011) has 100 , 000 images of digits ( 73 , 257 of which for training).

Original: SVHN includes color housenumber images which are classiﬁed in 10 classes; each class corresponds to a digit 0 to 9 .
Revised: SVHN includes color housenumber images which are classiﬁed in 10 classes; each class corresponds to a digit 0 to 9 .

Original: SVHN images are diverse (e.g., with respect to background, scale).
Revised: SVHN images are diverse (e.g., with respect to background, scale).

Original: • MNIST (LeCun et al., 1998) consists of images with handwritten digits.
Revised: • MNIST (LeCun et al., 1998) consists of images with handwritten digits.

Original: Each images depicts a single digit (annotated from 0 to 9 ) in a 28 ˆ 28 resolution.
Revised: Each images depicts a single digit (annotated from 0 to 9 ) in a 28 ˆ 28 resolution.

Original: The dataset includes 60 ,images for training.
Revised: The dataset includes 60 ,images for training.

Original: • Shoes (Yu & Grauman, 2014; Xie & Tu, 2015) consists of 50 , 000 images of shoes, where the edges of each shoe are extracted (Isola et al., 2017).
Revised: • Shoes (Yu & Grauman, 2014; Xie & Tu, 2015) consists of 50 , 000 images of shoes, where the edges of each shoe are extracted (Isola et al., 2017).

Original: • Handbags (Zhu et al., 2016; Xie & Tu, 2015) consists of more than 130 , 000 images of handbag items.
Revised: • Handbags (Zhu et al., 2016; Xie & Tu, 2015) consists of more than 130 , 000 images of handbag items.

Original: The edges have been computed for each image and used as conditional input to the generator (Isola et al., 2017).
Revised: The edges have been computed for each image and used as conditional input to the generator (Isola et al., 2017).

Revised: • Anime characters dataset (Jin et al., 2017) consists of anime characters that are generated based on speciﬁc attributes, e.g., hair color.

Revised: The public version used 5 contains annotations on the hair color and the eye color.

Revised: We consider 7 classes on the hair color and 6 classes on the eye color, with a total of 14 , 000 training images.

Original: All the images of CelebA, Cars196, Shoes and Handbags are resized to 64 ˆ 64 resolution.
Revised: All the images of CelebA, Cars196, Shoes and Handbags are resized to 64 ˆ 64 resolution.

Original: Architectures: The discriminator structure is left the same for each experiment, we focus only on the generator architecture.
Revised: Architectures: The discriminator structure is left the same for each experiment, we focus only on the generator architecture.

Original: All the architectures are based on two different generator schemes, i.e., the SNGAN (Miyato & Koyama, 2018) and the polynomial expansion of Chrysos et al.
Revised: All the architectures are based on two different generator schemes, i.e., the SNGAN (Miyato & Koyama, 2018) and the polynomial expansion of Chrysos et al.

Original:  that 
Revised: (2020) that does not include activation functions in the generator.

Original: The variants of the generator of SNGAN are described below:
Revised: The variants of the generator of SNGAN are described below:

Original: • SNGAN (Miyato & Koyama, 2018):
Revised: • SNGAN (Miyato & Koyama, 2018):

Original: The generator consists of a convolution, followed by three residual blocks.
Revised: The generator consists of a convolution, followed by three residual blocks.

Original: The discriminator is also based on successive residual blocks.
Revised: The discriminator is also based on successive residual blocks.

Original: The public implementation of SNGAN with conditional batch normalization (CBN) is used as the baseline.
Revised: The public implementation of SNGAN with conditional batch normalization (CBN) is used as the baseline.

Original: • SNGAN-MVP [proposed]: We convert the resnet-based generator of SNGAN to an MVP model.
Revised: • SNGAN-MVP [proposed]: We convert the resnet-based generator of SNGAN to an MVP model.

Original: To obtain MVP, the SNGAN is modiﬁed in two ways: a) the Conditional Batch Normalization (CBN) is converted into batch normalization (Ioffe & Szegedy, 2015), b) the injections of the two embeddings (from the inputs) are added after each residual block, i.e. the formula of (2).
Revised: To obtain MVP, the SNGAN is modiﬁed in two ways: a) the Conditional Batch Normalization (CBN) is converted into batch normalization (Ioffe & Szegedy, 2015), b) the injections of the two embeddings (from the inputs) are added after each residual block, i.e. the formula of (2).

Original: In other words, the generator is converted to a product of two-variable polynomials.
Revised: In other words, the generator is converted to a product of two-variable polynomials.

Original: • SNGAN-CONC : Based on SNGAN-MVP, we replace each Hadamard product with a concatenation.
Revised: • SNGAN-CONC : Based on SNGAN-MVP, we replace each Hadamard product with a concatenation.

Original: This implements the variant mentioned in sec.
Revised: This implements the variant mentioned in sec.

Original: D. • SNGAN-SPADE (Park et al., 2019):
Revised: D. • SNGAN-SPADE (Park et al., 2019):

Original: As described in sec.
Revised: As described in sec.

Original: F, SPADE is a polynomial with respect to the conditional variable z II .
Revised: F.1, SPADE is a polynomial with respect to the conditional variable z II .

Original: The generator of SNGAN-MVP is modiﬁed to perform the Hadamard product with respect to the conditional variable every time.
Revised: The generator of SNGAN-MVP is modiﬁed to perform the Hadamard product with respect to the conditional variable every time.

Original: The variants of the generator of Π -Net are described below:
Revised: The variants of the generator of Π -Net are described below:

Original: • Π -Net (Chrysos et al., 2020):
Revised: • Π -Net (Chrysos et al., 2020):

Original: The generator is based on a product of polynomials.
Revised: The generator is based on a product of polynomials.

Original: The ﬁrst polynomials use fully-connected connections, while the next few polynomials use cross-correlations.
Revised: The ﬁrst polynomials use fully-connected connections, while the next few polynomials use cross-correlations.

Original: The discriminator is based on the residual blocks of SNGAN.
Revised: The discriminator is based on the residual blocks of SNGAN.

Original: We stress out that the generator does not include any activation functions apart from a hyperbolic tangent in the output space for normalization.
Revised: We stress out that the generator does not include any activation functions apart from a hyperbolic tangent in the output space for normalization.

Original: The authors advocate that this exhibits the expressivity of the designed model.
Revised: The authors advocate that this exhibits the expressivity of the designed model.

Original: • Π -Net-SICONC :
Revised: • Π -Net-SICONC :

Original: The generator structure is based on Π -Net with two modiﬁcations: a) the Conditional Batch Normalization is converted into batch normalization (Ioffe & Szegedy, 2015), b) the second-input is concatenated with the ﬁrst (i.e., the noise) in the input of the generator.
Revised: The generator structure is based on Π -Net with two modiﬁcations: a) the Conditional Batch Normalization is converted into batch normalization (Ioffe & Szegedy, 2015), b) the second-input is concatenated with the ﬁrst (i.e., the noise) in the input of the generator.

Original: Thus, this is a single variable polynomial, i.e., a Π -Net, where the second-input is vectorized and concatenated with the ﬁrst.
Revised: Thus, this is a single variable polynomial, i.e., a Π -Net, where the second-input is vectorized and concatenated with the ﬁrst.

Original: This baseline implements the Π -Net described in sec.
Revised: This baseline implements the Π -Net described in sec.

Original: E. • MVP [proposed]: The generator of Π -Net is converted to an MVP model with two modiﬁcations: a) the Conditional Batch Normalization is converted into batch normalization (Ioffe & Szegedy, 2015), b) instead of having a Hadamard product with a single variable as in Π -Net, the formula with the two-variable input (e.g., (2)) is followed.
Revised: E. • MVP [proposed]: The generator of Π -Net is converted to an MVP model with two modiﬁcations: a) the Conditional Batch Normalization is converted into batch normalization (Ioffe & Szegedy, 2015), b) instead of having a Hadamard product with a single variable as in Π -Net, the formula with the two-variable input (e.g., (2)) is followed.

Original: • GAN-CONC : Based on MVP, each Hadamard product is replaced by a concatenation.
Revised: • GAN-CONC : Based on MVP, each Hadamard product is replaced by a concatenation.

Original: This implements the variant mentioned in sec.
Revised: This implements the variant mentioned in sec.

Original: D. • GAN-ADD : Based on MVP, each Hadamard product is replaced by an addition.
Revised: D. • GAN-ADD : Based on MVP, each Hadamard product is replaced by an addition.

Original: This modiﬁes (14) to x n “
Revised: This modiﬁes (14) to x n “

Original: [Equation]
Revised: [Equation]

Original: • SPADE (Park et al., 2019):
Revised: • SPADE (Park et al., 2019):

Original: As described in sec.
Revised: As described in sec.

Original: F, SPADE deﬁnes a polynomial with respect to the conditional variable z II .
Revised: F.1, SPADE deﬁnes a polynomial with respect to the conditional variable z II .

Original: The generator of Π -Net is modiﬁed to perform the Hadamard product with respect to the conditional variable every time.
Revised: The generator of Π -Net is modiﬁed to perform the Hadamard product with respect to the conditional variable every time.

Original: • SPADE-MVP [proposed]: This is a variant we develop to bridge the gap between SPADE and the proposed MVP.
Revised: • SPADE-MVP [proposed]: This is a variant we develop to bridge the gap between SPADE and the proposed MVP.

Original: Speciﬁcally, we augment the aforementioned SPADE twofold: a) the dense layers in the input space are converted into a polynomial with respect to the variable z
Revised: Speciﬁcally, we augment the aforementioned SPADE twofold: a) the dense layers in the input space are converted into a polynomial with respect to the variable z

Original: I and b) we also convert the polynomial in the output (i.e., the rightmost polynomial in the Fig.
Revised: I and b) we also convert the polynomial in the output (i.e., the rightmost polynomial in the Fig.

Original: 6 schematics) to a polynomial with respect to the variable z I .
Revised: 6 schematics) to a polynomial with respect to the variable z I .

Original: This model captures higher-order correlations of the variable z
Revised: This model captures higher-order correlations of the variable z

Original: I that SPADE did not not originally include.
Revised: I that SPADE did not not originally include.

Original: This model still includes single variable polynomials, however the input in each polynomial varies and is not only the conditional variable.
Revised: This model still includes single variable polynomials, however the input in each polynomial varies and is not only the conditional variable.

Original: The two baselines GAN-CONC and GAN-ADD capture only additive correlations, hence they cannot effectively model complex distributions without activation functions.
Revised: The two baselines GAN-CONC and GAN-ADD capture only additive correlations, hence they cannot effectively model complex distributions without activation functions.

Original: Nevertheless, they are added as a reference point to emphasize the beneﬁts of higher-order polynomial expansions.
Revised: Nevertheless, they are added as a reference point to emphasize the beneﬁts of higher-order polynomial expansions.

Original: An abstract schematic of the generators that are in the form of products of polynomials is depicted in
Revised: An abstract schematic of the generators that are in the form of products of polynomials is depicted in

Original: Fig.
Revised: Fig.

Original: Notice that the compared methods from the literature use polynomials of a single variable, while we propose a polynomial with an arbitrary number of inputs (e.g., two-input shown in the schematic).
Revised: Notice that the compared methods from the literature use polynomials of a single variable, while we propose a polynomial with an arbitrary number of inputs (e.g., two-input shown in the schematic).

Original: c
Revised: c

Revised: Poly

Original: [Figure]
Revised: [Figure]

Original: 1 -var 1 -var 1 -var 1 -var
Revised: Poly

Original: [Figure]
Revised: [Figure]

Original: c
Revised: Poly [Figure] Poly

Original: [Figure]
Revised: [Figure]

Original: ∼ 𝑧 
Revised: x 𝑧 𝐼 ∼ 1 -var 1 -var 1 -var 1 -var

Original: [Figure]
Revised: [Figure]

Original:  ∼ 
Revised: c 𝑧 𝐼 ∼ 𝑧 𝐼𝐼 ∼ 𝐵𝑀𝑊,… or [Figure] Poly

Original: [Figure]
Revised: [Figure]

Original: x 1 -var 1 -var 1 -var 1 -var
Revised: x 1 -var 1 -var 1 -var 1 -var

Original: [Figure]
Revised: [Figure]

Original: c 
Revised: c 𝑧 𝐼 ∼ 𝑧 𝐼𝐼 ∼ 𝐵𝑀𝑊,… or Dense Poly [Figure]

Revised: Poly

Revised: [Figure]

Revised: Poly

Original: [Figure]
Revised: [Figure]

Original:  1 -var 1 -var 1 -var 
Revised: x 1 -var 1 -var 1 -var (

Original: [Figure]
Revised: [Figure]

Original: c 𝑧
Revised: [Figure] [Figure] al.,

Original: [Figure]
Revised: [Figure]

Original:  ∼ 
Revised: ) c 𝑧 𝐼 ∼ 𝑧 𝐼𝐼 ∼ 𝐵𝑀𝑊,… or

Original: [Figure]
Revised: [Figure]

Original:  2 -var 2 -var 2 -var
Revised: x 2 -var 2 -var 2 -var

Original: [Figure]
Revised: [Figure]

Original: var
Revised: var

Original: [Figure]
Revised: [Figure]

Original: to the conditional variable z II . This is substantially different from the polynomial with multiple-input variables, i.e., MVP. Two additional differences are that (i) SPADE is motivated as a spatially-adaptive method (i.e., for continuous conditional variables), while MVP can be used both for discrete and for continuous type variables, (ii) there is no polynomial in the dense layers in the SPADE.
Revised: Two additional differences are that (i) SPADE is motivated as a spatially-adaptive method (i.e., for continuous conditional variables), while MVP can be used both for discrete and for continuous type variables, (ii) there is no polynomial in the dense layers in the SPADE.

Original: However, as illustrated in Π -Net converting the dense layers into a higher-order polynomial can further boost the performance.
Revised: However, as illustrated in Π -Net converting the dense layers into a higher-order polynomial can further boost the performance.

Original: (d) The proposed generator structure.
Revised: (d) The proposed generator structure.

Original: Implementation details of MVP:
Revised: Implementation details of MVP:

Original: Throughout this work, we reserve the symbol z II for the conditional input (e.g., a class label).
Revised: Throughout this work, we reserve the symbol z II for the conditional input (e.g., a class label).

Original: In each polynomial, we reduce further the parameters by using the same embedding for the conditional variables.
Revised: In each polynomial, we reduce further the parameters by using the same embedding for the conditional variables.

Original: That is expressed as:
Revised: That is expressed as:

Original: [Equation]
Revised: [Equation]

Original:  for n “ 2 , . . .
Revised: (19) for n “ 2 , . . .

Original: , N .
Revised: , N .

Original: Equivalently, that would be A r n,II s “ A r 1 ,II s in (14).
Revised: Equivalently, that would be A r n,II s “ A r 1 ,II s in (14).

Original: Additionally, Nested-MVP performed better in our preliminary experiments, thus we use (14) to design each polynomial.
Revised: Additionally, Nested-MVP performed better in our preliminary experiments, thus we use (14) to design each polynomial.

Original: Given the aforementioned sharing, the N th order expansion is described by:
Revised: Given the aforementioned sharing, the N th order expansion is described by:

Original: [Equation]
Revised: [Equation]

Original: for n “ 2 , . . .
Revised: for n “ 2 , . . .

Original: , N .
Revised: , N .

Original: Lastly, the factor A r 1 ,II s is a convolutional layer in the case of continuous conditional input, while it is a fully-connected layer in the case of discrete conditional input.
Revised: Lastly, the factor A r 1 ,II s is a convolutional layer in the case of continuous conditional input, while it is a fully-connected layer in the case of discrete conditional input.

Original: H A DDITIONAL EXPERIMENTS
Revised: H A DDITIONAL EXPERIMENTS

Original: Additional experiments and visualizations are provided in this section.
Revised: Additional experiments and visualizations are provided in this section.

Original: Additional visualizations for class-conditional generation are provided in sec.
Revised: Additional visualizations for class-conditional generation are provided in sec.

Original: H.1.
Revised: H.1.

Original: An  experiment with mixed conditional input is conducted in sec.
Revised: An additional experiment with class-conditional generation with SVHN digits is performed in sec.

Original: H.5.
Revised: H.2.

Original: Additional experiments with class-conditional generation with SVHN digits and translation of MNIST to SVHN digits are conducted in sec.H.2 and sec.
Revised: An experiment that learns the translation of MNIST to SVHN digits is conducted in sec.

Original: H.3 respectively.
Revised: H.3.

Original: Two experimentstranslating edges to images are conducted in sec.
Revised: To explore further the image-to-image translation, two additional experiments are conducted in sec.

Original: H.4.
Revised: H.4.

Revised: An attribute-guided generation is performed in sec.

Revised: H.5 to illustrate the beneﬁt of our framework with respect to multiple, discrete conditional inputs.

Revised: This is further extended in sec.

Revised: H.6, where an experiment with mixed conditional input is conducted.

Revised: Finally, an additional diversity-inducing regularization term is used to assess whether it can further boost the diversity the synthesized images in sec.

Original:  H. A DDITIONAL VISUALIZATIONS IN CLASS - CONDITIONAL GENERATION
Revised: H.7. H. A DDITIONAL VISUALIZATIONS IN CLASS - CONDITIONAL GENERATION

Original: In Fig.
Revised: In Fig.

Original: 7 the qualitative results of the compared methods in class-conditional generation on CIFAR are shared.
Revised: 7 the qualitative results of the compared methods in class-conditional generation on CIFAR are shared.

Original: Both the generator of SNGAN and ours have activation functions in this experiment.
Revised: Both the generator of SNGAN and ours have activation functions in this experiment.

Original: [Figure]
Revised: [Figure]

Original: In Fig.
Revised: In Fig.

Original: 8 samples from the baseline Π -Net (Chrysos et al., 2020) and our method are depicted for the class-conditional generation on CIFAR10.
Revised: 8 samples from the baseline Π -Net (Chrysos et al., 2020) and our method are depicted for the class-conditional generation on CIFAR10.

Original: The images have a substantial difference.
Revised: The images have a substantial difference.

Original: Similarly, in Fig.
Revised: Similarly, in Fig.

Original: 9 a visual comparison between Π -Net and MVP is exhibited in Cars196 dataset.
Revised: 9 a visual comparison between Π -Net and MVP is exhibited in Cars196 dataset.

Original: To our knowledge, no framework in the past has demonstrated such expressivity; MVP synthesizes images that approximate the quality of synthesized images from networks with activation functions.
Revised: To our knowledge, no framework in the past has demonstrated such expressivity; MVP synthesizes images that approximate the quality of synthesized images from networks with activation functions.

Original: In Fig.
Revised: In Fig.

Original: 10, an inter-class interpolation of various compared methods in CIFAR10 are visualized.
Revised: 10, an inter-class interpolation of various compared methods in CIFAR10 are visualized.

Original: The illustrations of the intermediate images in SNGAN-CONC and SNGAN-ADD are either blurry or not realistic.
Revised: The illustrations of the intermediate images in SNGAN-CONC and SNGAN-ADD are either blurry or not realistic.

Original: On the contrary, in SPADE and MVP the higher-order polynomial expansion results in more realistic intermediate images.
Revised: On the contrary, in SPADE and MVP the higher-order polynomial expansion results in more realistic intermediate images.

Original: Nevertheless, MVP results in sharper shapes and images even in the intermediate results when compared to SPADE.
Revised: Nevertheless, MVP results in sharper shapes and images even in the intermediate results when compared to SPADE.

Original: H. C LASS - CONDITIONAL GENERATION ON HOUSE DIGITS
Revised: H. C LASS - CONDITIONAL GENERATION ON HOUSE DIGITS

Original: An experiment on class-conditional generation with SVHN is conducted below.
Revised: An experiment on class-conditional generation with SVHN is conducted below.

Original: SVHN images include (substantial) blur or other distortions, which insert noise in the distribution to be learned.
Revised: SVHN images include (substantial) blur or other distortions, which insert noise in the distribution to be learned.

Original: In addition, some images contain contain a central digit (i.e., based on which the class is assigned), andpartial visibility of other digits.
Revised: In addition, some images contain contain a central digit (i.e., based on which the class is assigned), and

Original: Therefore, the generation of digits of SVHN is challenging for a generator without activation functions between the layers.

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Revised: partial visibility of other

Revised: Therefore, generation of SVHN is challenging

Original: Our framework, e.g., equation 14, does not include any activation functions.
Revised: Our framework, e.g., equation 14, does not include any activation functions.

Original: To verify the expressivity of our framework, we maintain the same setting for this experiment.
Revised: To verify the expressivity of our framework, we maintain the same setting for this experiment.

Original: Particularly, the generator does not have activation functions between the layers ; there is only a hyperbolic tangent in the output space for normalization.
Revised: Particularly, the generator does not have activation functions between the layers ; there is only a hyperbolic tangent in the output space for normalization.

Original: The generator receives a noise sample and a class as input, i.e., it is a class-conditional polynomial generator. 
Revised: The generator receives a noise sample and a class as input, i e., it is a class-conditional polynomial generator. [Figure]

Original: The results in Fig.
Revised: The results in Fig.

Original: 12(b) illustrate that despite the noise, MVP learns the distribution.
Revised: 12(b) illustrate that despite the noise, MVP learns the distribution.

Original: As mentioned in the main paper, our formulation enables both inter-class and intra-class interpolations naturally.
Revised: As mentioned in the main paper, our formulation enables both inter-class and intra-class interpolations naturally.

Original: In the inter-class interpolation the noise z
Revised: In the inter-class interpolation the noise z

Original: I is ﬁxed, while the class z II is interpolated.
Revised: I is ﬁxed, while the class z II is interpolated.

Original: In Fig.
Revised: In Fig.

Original: 12(d)
Revised: 12(d)

Original: [Figure]
Revised: several inter-class interpolations are visualized. The visualization exhibits that our framework is able to synthesize realistic images even with inter-class interpolations.

Original: H. T RANSLATION OF MNIST DIGITS TO SVHN DIGITS
Revised: H. T RANSLATION OF MNIST DIGITS TO SVHN DIGITS

Original: An experiment on image translation from the domain of binary digits to house numbers is conducted below.
Revised: An experiment on image translation from the domain of binary digits to house numbers is conducted below.

Original: The images of MNIST are used as the source domain (i.e., the conditional variable z II ), while  the images of SVHN are used as the target domain.
Revised: The images of MNIST are used as the source domain (i.e., the conditional variable z II ), while [Figure] [Figure] the images of SVHN are used as the target domain.

Original: The correspondence of the source to the target domain is assumed to be many-to-many, i.e., each MNIST digit can synthesize multiple SVHN images.
Revised: The correspondence of the source to the target domain is assumed to be many-to-many, i e., each MNIST digit can synthesize multiple SVHN images.

Original: No additional loss is used, the setting of continuous conditional input from sec.
Revised: No additional loss is used, the setting of continuous conditional input from sec.

Original: 4.2 is used.
Revised: 4.2 is used.

Original: [Figure]

Original: [Figure]

Original: The images in Fig.
Revised: The images in Fig.

Original: 13 illustrate that MVP can translate MNIST digits into SVHN digits.
Revised: 13 illustrate that MVP can translate MNIST digits into SVHN digits.

Original: Additionally, for each source digit, there is a signiﬁcant variation in the synthesized images.
Revised: Additionally, for each source digit, there is a signiﬁcant variation in the synthesized images.

Original:  T RANSLATION OF EDGES TO IMAGES
Revised: H. T RANSLATION OF EDGES TO IMAGES

Original: An additional experiment on translation is conducted, where the source domain depicts edges and the target domain is the output image.
Revised: An additional experiment on translation is conducted, where the source domain depicts edges and the target domain is the output image.

Original: Speciﬁcally, the tasks of edges-to-handbags (on Handbags dataset) and edges-to-shoes (on Shoes dataset) have been selected Isola et al.
Revised: Speciﬁcally, the tasks of edges-to-handbags (on Handbags dataset) and edges-to-shoes (on Shoes dataset) have been selected Isola et al.

Original: In this experiment, the MVP model of sec.
Revised: In this experiment, the MVP model of sec.

Original: 4.2 is utilized, i.e., a generator without activation functions between the layers.
Revised: 4.2 is utilized, i.e., a generator without activation functions between the layers.

Original: The training is conductedonly using  the adversarial loss.
Revised: The training is conducted using only the adversarial loss.

Original: Visual results for both the case of edges-to-handbags and edges-to-shoes are depicted in Fig.
Revised: Visual results for both the case of edges-to-handbags and edges-to-shoes are depicted in Fig.

Original: The ﬁrst row depicts the conditional input z II , i.e., an edge, while the rows 2-6 depict the synthesized images.
Revised: The ﬁrst row depicts the conditional input z II , i.e., an edge, while the rows 2-6 depict the synthesized images.

Original: Note that in both the case of handbags and shoes there is signiﬁcant variation in the synthesized images, while they follow the edges provided as input.
Revised: Note that in both the case of handbags and shoes there is signiﬁcant variation in the synthesized images, while they follow the edges provided as input.

Original: [Figure]
Revised: [Figure]

Original: H. M ULTIPLE  CONDITIONAL INPUTS
Revised: H. M ULTIPLE , DISCRETE CONDITIONAL INPUTS

Original: Frequently, more than one type of input conditional inputs are available.
Revised: Frequently, more than one type of input conditional inputs are available.

Original: Our formulation can be extended beyond two input variables (sec. C); we experimentally verify this case.
Revised: Our formulation can be extended beyond two input variables (sec. C); we experimentally verify this case.

Original: We select the case of two conditional input variables: one discrete (class label) and one continuous (low-resolution image).
Revised: The task selected is attribute-guided generation trained on images of Anime characters. Each image is annotated with respect to the color of the eyes ( 6 combinations) and the color of the hair ( 7 combinations). Since SPADE only accepts a single conditional variable, we should concatenate the two attributes in a single variable. We tried simply concatenating the attributes directly, but this did not work well.

Revised: Instead, we can use the total number of combinations, which is the product of the individual attribute combinations, i.e., in our case the total number of combinations is 42 .

Revised: Obviously, this causes ‘few’ images to belong in each unique combination, i.e., there are 340 images on average that belong to each combination.

Revised: On the contrary, there are 2380 images on average for each eye color.

Revised: SPADE and Π -Net are trained by using the two attributes in a single combination, while in our case, we consider the multiple conditional variable setting.

Revised: In each case, only the generator differs depending on the compared method.

Revised: In Fig.

Revised: 15 few indicative images are visualized for each method; each row depicts a single combination of attributes, i.e., hair and eye color.

Revised: Notice that SPADE results in a single image per combination, while in Π -Net-SINCONC there is considerable repetition in each case.

Revised: The single image in SPADE can be explained by the lack of higher-order correlations with respect to the noise variable

Revised: z

Revised: I .

Revised: In addition to the diversity of the images per combination, an image from every combination is visualized in Fig.

Revised: MVP synthesizes more realistic images than the compared methods of Π -NetSINCONC and SPADE.

Revised: [Figure]

Revised: [Figure]

Revised: H. M ULTIPLE CONDITIONAL INPUTS WITH MIXED CONDITIONAL VARIABLES

Revised: We extend the previous experiment with multiple conditional variables to the case of mixed conditional variables, i.e., there is one discrete and one continuous conditional variable.

Revised: The discrete conditional variable captures the class label, while the continuous conditional variable captures the low-resolution image.

Original: Thus, the task is class-conditional super-resolution.
Revised: Thus, the task is class-conditional super-resolution.

Original: We use the experimental details of sec.
Revised: We use the experimental details of sec.

Original: 4.2 in super-resolution 8 ˆ .
Revised: 4.2 in super-resolution 8 ˆ .

Original: In Fig.
Revised: In Fig.

Original: 15, we visualize how for each low-resolution image the results differ depending on the randomly sampled class label.
Revised: 17, we visualize how for each low-resolution image the results differ depending on the randomly sampled class label.

Original: The FID in this case is 53 . 63 , which is similar to the previous two cases.
Revised: The FID in this case is 53 . 63 , which is similar to the previous two cases.

Original: Class-conditional super-resolution (or similar tasks with multiple conditional inputs) can be of interest to the community and MVP results in high-dimensional images with large variance. 
Revised: Class-conditional super-resolution (or similar tasks with multiple conditional inputs) can be of interest to the community and MVP results in high-dimensional images with large variance. H. I MPROVE DIVERSITY WITH REGULARIZATION

Revised: As emphasized in sec.

Revised: I, various methods have been utilized for synthesizing more diverse images in conditional image generation tasks.

Revised: A reasonable question is whether our method can be used in conjunction with such methods, since it already synthesizes diverse results.

Revised: Our hypothesis is that when MVP is used in conjunction with any diversity-inducing technique, it will further improve the diversity of the synthesized images.

Revised: To assess the hypothesis, we conduct an experiment on edges to images that is a popular benchmark in such diverse generation tasks (Zhu et al., 2017b; Yang et al., 2019).

Revised: The plug-n-play regularization term of Yang et al.

Revised: (2019) is selected and added to the GAN loss during the training.

Revised: The objective of the regularization term

Revised: L reg is to maximize the following term:

Revised: [Equation]

Revised: [Equation]

Revised: where τ is a predeﬁned constant, z I,1 , z I,2 are different noise samples.

Revised: The motivation behind this term lies in encouraging the generator to produce outputs that differ when the input noise samples differ.

Revised: In our experiments, we follow the implementation of the original paper with τ “ 10 .

Revised: The regularization loss of equation 21 is added to the GAN loss; the architecture of the generator remains similar to sec.

Revised: H.4.

Revised: The translation task is edges-to-handbags (on Handbags dataset) and edges-to-shoes (on Shoes dataset).

Revised: In Fig.

Revised: 18 the synthesized images are depicted.

Revised: The regularization loss causes more diverse images to be synthesized (i.e., when compared to the visualization of Fig.that was trained using only the adversarial loss).

Revised: For instance, in both the shoes and the handbags, new shades of blue are now synthesized, while yellow handbags can now be synthesized.

Revised: The empirical results validate the hypothesis that our model can be used in conjunction with diversity regularization losses in order to improve the results.

Revised: Nevertheless, the experiment in sec.

Revised: H. indicates that a regularization term is not necessary to synthesize images that do not ignore the noise as feed-forward generators had previously.

Revised: I D IFFERENCE OF MVP FROM OTHER DIVERSE GENERATION TECHNIQUES

Revised: One challenge that often arises in conditional data generation is that one of the variables gets ignored by the generator (Isola et al., 2017).

Revised: This has been widely acknowledged in the literature, e.g., Zhu et al.

Revised: (2017b) advocates that it is hard to utilize a simple architecture, like Isola et al.

Revised: (2017), with noise.

Revised: A similar conclusion is drawn in InfoGAN (Chen et al., 2016) where the authors explicitly mention that additional losses are required, otherwise the generator is ‘free to ignore’ the additional variables.

Revised: To mitigate this, a variety of methods have been developed.

Revised: We summarize the most prominent methods from the literature, starting from image-to-image translation methods:

Revised: • BicycleGAN (Zhu et al., 2017b) proposes a framework that can synthesize diverse images in image-to-image translation.

Revised: The framework contains 2 encoders, 1 decoder anddiscriminators.

Revised: This results in multiple loss terms (e.g., eq.9 of the paper).

Revised: Interestingly, the authors utilize a separate training scheme for the encoder-decoder and the second encoder

Revised: [Figure]

Revised: as training together ’hides the information of the latent code without learning meaningful modes’.

Revised: • Almahairi et al.

Revised: (2018) augment the deterministic mapping of CycleGAN (Zhu et al., 2017a) with a marginal matching loss.

Revised: The framework learns diverse mappings utilizing the additional encoders.

Revised: The framework includes 4 encoders, 2 decoders and 2 discriminators.

Revised: • MUNIT (Huang et al., 2018) focuses on diverse generation in unpaired image-to-image translation.

Revised: MUNIT demonstrates impressive translation results, while the inverse translation is also learnt simultaneously.

Revised: That is, in case of edges-to-shoes, the translation shoes-toedges is also learnt during the training.

Revised: The mapping learnt comes at the cost of multiple network modules.

Revised: Particularly, MUNIT includes 2 encoders, 2 decoders, 2 discriminators for learning.

Revised: This also results in multiple loss terms (e.g., eq.5 of the paper) along with additional hyper-parameters and network parameters.

Revised: • Drit++ (Lee et al., 2020) extends unpaired image-to-image translation with disentangled representation learning, while they allow multi-domain image-to-image translations.

Revised: Drit++ uses 4 encoders, 2 decoders, 2 discriminators for learning.

Revised: Similarly to the previous methods, this results in multiple loss terms (e.g., eq.6-7 of the paper) and additional hyper-parameters.

Revised: • Choi et al.

Revised: (2020) introduce a method that supports multiple target domains.

Revised: The method includes four modules: a generator, a mapping network, a style encoder and a discriminator

Revised: All modules (apart from the generator) include domain-speciﬁc sub-networks in case of multiple target domains.

Revised: To ensure diverse generation, Choi et al.

Revised: (2020) utilize a regularization loss (i.e., eq. 3 of the paper), while their ﬁnal objective consists of multiple loss terms.

Revised: The aforementioned frameworks contain additional network modules for training, which also results in additional hyper-parameters in the loss-function and the network architecture.

Revised: Furthermore, the frameworks focus exclusively on image-to-image translation and not all conditional generation cases, e.g., they do not tackle class-conditional or attribute-based generation.

Revised: An interesting technique for diverse, class-conditional generation is the self-conditional GAN of Liu et al.

Revised: The method conditions the generator with pseudo-labels that are automatically derived from clustering on the feature space of the discriminator.

Revised: This enables the generator to synthesize more diverse samples.

Revised: This method is orthogonal to our, i.e., the generator of Liu et al.

Revised: (2020) can be replaced with MVP.

Revised: Using regularization terms in the loss function has been an alternative way to achieve diverse generation.

Revised: Mao et al.

Revised: ; Yang et al.

Revised: (2019) propose simple regularization terms that can be plugged into any architecture to encourage diverse generation.

Revised: Lee et al.

Revised: (2019) propose two variants of a regularization term, with the ‘more stable variant’ requiring additional network modules.

Revised: We emphasize that our method can be used in conjunction with many of the aforementioned techniques to obtain more diverse examples.

Revised: We demonstrate that this is possible in an experiment in sec.

Revised: H.7.