Original: Generative Visual Prompt: Unifying Distributional Control of Pre-Trained Generative Models
Revised: Generative Visual Prompt: Unifying Distributional Control of Pre-Trained Generative Models

Original: Anonymous Author(s)
Revised: Anonymous Author(s)

Original: AfﬁliationAddress email
Revised: AfﬁliationAddress email

Original: Abstract
Revised: Abstract

Original: Generative models (e.g., GANs, VAEs) learn the underlying data distribution inan unsupervised manner.
Revised: Generative models (e.g., GANs, VAEs) learn the underlying data distribution inan unsupervised manner.

Original: However, many applications of interest require samplingfrom a speciﬁc region of the generative model’s output space or evenly over arange of characteristics.
Revised: However, many applications of interest require samplingfrom a speciﬁc region of the generative model’s output space or evenly over arange of characteristics.

Original: To allow efﬁcient sampling in these scenarios, we propose
Revised: To allow efﬁcient sampling in these scenarios, we propose

Original: Prompt Generation (PromptGen), a framework to achieve distributional controlover pre-trained generative models by incorporating knowledge of arbitrary offthe-shelf models.
Revised: Prompt Generation (PromptGen), a framework to achieve distributional controlover pre-trained generative models by incorporating knowledge of arbitrary offthe-shelf models.

Original: PromptGen deﬁnes control as an energy-based model (EBM)and samples images in a feed-forward manner by approximating the EBM withinvertible neural networks, avoiding optimization at inference.
Revised: PromptGen deﬁnes control as an energy-based model (EBM)and samples images in a feed-forward manner by approximating the EBM withinvertible neural networks, avoiding optimization at inference.

Original: We demonstratehow PromptGen can control several generative models using various off-the-shelfmodels: (1) with the CLIP model, PromptGen can sample images guided by text,(2) with image classiﬁers, PromptGen can de-bias generative models across a set ofattributes, and (3) with inverse graphics models, PromptGen can sample images ofthe same identity in different poses.
Revised: We demonstratehow PromptGen can control several generative models using various off-the-shelfmodels: (1) with the CLIP model, PromptGen can sample images guided by text,(2) with image classiﬁers, PromptGen can de-bias generative models across a set ofattributes, and (3) with inverse graphics models, PromptGen can sample images ofthe same identity in different poses.

Original: (4) Finally, PromptGen reveals that the CLIPmodel shows “reporting bias” when used as control, and PromptGen can furtherde-bias this controlled distribution in an iterative manner.
Revised: (4) Finally, PromptGen reveals that the CLIPmodel shows “reporting bias” when used as control, and PromptGen can furtherde-bias this controlled distribution in an iterative manner.

Original:  Introduction
Revised:  Introduction

Original: Generative models (e.g., GANs, VAEs) learn the underlying data distribution in high-dimensionalspaces and achieve promising performance on image synthesis [4, 73, 35, 21].
Revised: Generative models (e.g., GANs, VAEs) learn the underlying data distribution in high-dimensionalspaces and achieve promising performance on image synthesis [4, 74, 35, 21].

Original: Though being wellpraised, they still face two main criticisms.
Revised: Though being wellpraised, they still face two main criticisms.

Original: First, since generative models are typically trained in anunsupervised way, they lack controllability, meaning that it is unclear how to sample from a speciﬁcregion of the space.
Revised: First, since generative models are typically trained in anunsupervised way, they lack controllability, meaning that it is unclear how to sample from a speciﬁcregion of the space.

Original: Second, generative models are prone to inherit the imbalance and bias of trainingdata [58, 30].
Revised: Second, generative models are prone to inherit the imbalance and bias of trainingdata [58, 30].

Original: For instance, StyleGAN2 is more likely to produce images of white individuals, see
Revised: For instance, StyleGAN2 is more likely to produce images of white individuals, see

Original: Figure 1(e).
Revised: Figure 1(e).

Original: Previous works have studied these challenges separately, and typical methods includeediting of “style” codes [22, 66, 30] and explicit conditions [41, 67].
Revised: Previous works have studied these challenges separately, and typical methods includeediting of “style” codes [22, 66, 30] and explicit conditions [41, 67].

Original: However, these methods areeither model-dependent (i.e., requiring a well-structured style space) or label-intensive (i.e., requiringall training samples to be labeled for explicit conditions), limiting their generality and practical use.
Revised: However, these methods areeither model-dependent (i.e., requiring a well-structured style space) or label-intensive (i.e., requiringall training samples to be labeled for explicit conditions), limiting their generality and practical use.

Original: To address the above challenges, this paper advocates a uniﬁed formulation, distributional controlof generative models , which enables controllability by incorporating the knowledge of off-the-shelfmodels (e.g., CLIP [56], classiﬁers, or inverse graphics models [16]).
Revised: To address the above challenges, this paper advocates a uniﬁed formulation, distributional controlof generative models , which enables controllability by incorporating the knowledge of off-the-shelfmodels (e.g., CLIP [56], classiﬁers, or inverse graphics models [16]).

Original: Based on this uniﬁed view, wepropose to learn distributions in the latent space of a pre-trained generative model while keeping thepre-trained parameters ﬁxed.
Revised: Based on this uniﬁed view, wepropose to learn distributions in the latent space of a pre-trained generative model while keeping thepre-trained parameters ﬁxed.

Original: Given its conceptual similarity to prompt learning [40, 82, 72, 83, 29],we term our framework as Prompt Generation (PromptGen).
Revised: Given its conceptual similarity to prompt learning [40, 84, 72, 85, 29],we term our framework as Prompt Generation (PromptGen).

Original: PromptGen requires no training data ,and the only supervision comes from off-the-shelf models that help deﬁne the control.
Revised: PromptGen requires no training data ,and the only supervision comes from off-the-shelf models that help deﬁne the control.

Original: Speciﬁcally,
Revised: Speciﬁcally,

Original: PromptGen allows the user to deﬁne controls using an energy-based model (EBM) approximated by an invertible neural network (INN).
Revised: PromptGen allows the user to deﬁne controls using an energy-based model (EBM) approximated by an invertible neural network (INN).

Original: Unlike methods that require optimization at inference in
Revised: Unlike methods that require optimization at inference in

Original: EBM sampling [47, 14, 43, 50], PromptGen samples images in a feed-forward manner, which ishighly efﬁcient.
Revised: EBM sampling [47, 14, 43, 50], PromptGen samples images in a feed-forward manner, which ishighly efﬁcient.

Original: Moreover, PromptGen stands alone at inference, meaning that we can discard theoff-the-shelf models, which deﬁne the control, after training.
Revised: Moreover, PromptGen stands alone at inference, meaning that we can discard theoff-the-shelf models, which deﬁne the control, after training.

Original: We illustrate the beneﬁts of PromptGen with several experiments.
Revised: We illustrate the beneﬁts of PromptGen with several experiments.

Original: Figure 1 demonstrates our mainﬁndings with StyleGAN2 trained on FFHQ [34] without any labels, while we also show results for
Revised: Figure 1 demonstrates our mainﬁndings with StyleGAN2 trained on FFHQ [34] without any labels, while we also show results for

Original: StyleNeRF
Revised: StyleNeRF

Original: [21] and NVAE [73] on non-facial data in Section 4.
Revised: [21] and NVAE [74] on non-facial data in Section 4.

Original: Figure 1(a) illustrates how wecan sample from StyleGAN2 based on text descriptions such as “a photo of a baby”.
Revised: Figure 1(a) illustrates how wecan sample from StyleGAN2 based on text descriptions such as “a photo of a baby”.

Original: Figure 1(b)shows that PromptGen can leverage a race classiﬁer (potentially trained on a different dataset) tosample uniformly across all races, de-biasing the pre-trained StyleGAN2.
Revised: Figure 1(b)shows that PromptGen can leverage a race classiﬁer (potentially trained on a different dataset) tosample uniformly across all races, de-biasing the pre-trained StyleGAN2.

Original: Figure 1(d) illustrates that
Revised: Figure 1(d) illustrates that

Original: PromptGen can generate images of the same identity in different poses, guided by a pose regressor.
Revised: PromptGen can generate images of the same identity in different poses, guided by a pose regressor.

Original: Finally, it is worth pointing out that PromptGen not only offers generality for algorithmic designand modularity for control composition, but also enables iterative controls when some controls arecontingent on others.
Revised: Finally, it is worth pointing out that PromptGen not only offers generality for algorithmic designand modularity for control composition, but also enables iterative controls when some controls arecontingent on others.

Original: For instance, one may train a text-controlled distribution and then de-bias thisdistribution.
Revised: For instance, one may train a text-controlled distribution and then de-bias thisdistribution.

Original: To achieve this, we view the composition of the INN and the generative model as a new“generative model” to be controlled.
Revised: To achieve this, we view the composition of the INN and the generative model as a new“generative model” to be controlled.

Original: Figure 1(f) illustrates that PromptGen reveals “reporting bias” ofthe CLIP model [56], where “without makeup” is – perhaps surprisingly – positively correlated withfemale, and Figure 1(c) shows that PromptGen can further mitigate this bias with iterative control.
Revised: Figure 1(f) illustrates that PromptGen reveals “reporting bias” ofthe CLIP model [56], where “without makeup” is – perhaps surprisingly – positively correlated withfemale, and Figure 1(c) shows that PromptGen can further mitigate this bias with iterative control.

Original: [Table]
Revised: [Table]

Original:  Related Work
Revised:  Related Work

Original: Over the past few years, generative models have gained the ability to generate images with high visualquality.
Revised: Over the past few years, generative models have gained the ability to generate images with high visualquality.

Original: A few of the most widely used methods include generative adversarial networks (GANs) [18],
Revised: A few of the most widely used methods include generative adversarial networks (GANs) [18],

Original: VAE [38], invertible neural networks [11, 12], and diffusion models [24, 69].
Revised: VAE [38], invertible neural networks [11, 12], and diffusion models [24, 69].

Original: In particular, generativemodels trained on large amounts of unlabeled data, e.g., BigGAN [4], StyleGANs [34, 35, 33, 62],
Revised: In particular, generativemodels trained on large amounts of unlabeled data, e.g., BigGAN [4], StyleGANs [34, 35, 33, 62],

Original: Glow [37], and diffusion models [49, 10], achieve promising image synthesis results.
Revised: Glow [37], and diffusion models [49, 10], achieve promising image synthesis results.

Original: Despite their success, controllability and de-biasing are still two fundamental challenges generativemodels face.
Revised: Despite their success, controllability and de-biasing are still two fundamental challenges generativemodels face.

Original: For controllability, existing methods include explicit conditioning at training [41, 67] or local editing of the learned representation, e.g, “style” codes [1, 54, 44, 66, 77].
Revised: For controllability, existing methods include explicit conditioning at training [41, 67] or local editing of the learned representation, e.g, “style” codes [1, 54, 44, 66, 79].

Original: For de-biasing,existing methods include local editing of “style” codes [58, 30] and importance sampling for eithertraining [20] or inference [28].
Revised: For de-biasing,existing methods include local editing of “style” codes [58, 30] and importance sampling for eithertraining [20] or inference [28].

Original: Existing works study these problems separately, each requiring aspeciﬁc design for the task studied.
Revised: Existing works study these problems separately, each requiring aspeciﬁc design for the task studied.

Original: On the contrary, PromptGen is a uniﬁed framework for arbitrarycontrols deﬁned by off-the-shelf models.
Revised: On the contrary, PromptGen is a uniﬁed framework for arbitrarycontrols deﬁned by off-the-shelf models.

Original: The beneﬁts of unifying tasks have been shown by a recenttrend of works in multiple research areas across vision and language [57, 46, 61, 78, 59].
Revised: The beneﬁts of unifying tasks have been shown by a recenttrend of works in multiple research areas across vision and language [57, 46, 61, 80, 59].

Original: Moreover,one can ﬁne-tune generative models [17] for domain adaptation, which is orthogonal to PromptGen:since PromptGen maps a generative model to another generative model (Algorithm 1), ﬁne-tuningcan be applied before or after PromptGen training.
Revised: Moreover,one can ﬁne-tune generative models [17] for domain adaptation, which is orthogonal to PromptGen:since PromptGen maps a generative model to another generative model (Algorithm 1), ﬁne-tuningcan be applied before or after PromptGen training.

Original: We leave this exploration to future studies.
Revised: We leave this exploration to future studies.

Original: Previous methods usually sample from EBM [39] with Markov Chain Monte Carlo (MCMC) [71, 74,14, 19, 13, 43].
Revised: Previous methods usually sample from EBM [39] with Markov Chain Monte Carlo (MCMC) [71, 75,14, 19, 13, 43].

Original: Among them, plug-and-play generative models (PPGMs) [47] and LACE [50] deﬁnelatent-space EBMs.
Revised: Among them, plug-and-play generative models (PPGMs) [47] and LACE [50] deﬁnelatent-space EBMs.

Original: However, MCMC requires inference-time optimization, which is inefﬁcient andrequires the off-the-shelf models to be available at inference; this is also the criticism for diffusionmodels, regardless of being gradient-guided [49, 65] or not [24, 25, 48].
Revised: However, MCMC requires inference-time optimization, which is inefﬁcient andrequires the off-the-shelf models to be available at inference; this is also the criticism for diffusionmodels, regardless of being gradient-guided [49, 65] or not [24, 25, 48].

Original: In contrast, PromptGenachieves efﬁcient, feed-forward sampling.
Revised: In contrast, PromptGenachieves efﬁcient, feed-forward sampling.

Original: Table 1 shows a comparison with previous methods.
Revised: Table 1 shows a comparison with previous methods.

Original: Our
Revised: Our

Original: INN training is similar to that proposed by [52] to sample from physical systems, which is later usedby [75] to solve inverse problems with two composed INNs.
Revised: INN training is similar to that proposed by [52] to sample from physical systems, which is later usedby [77] to solve inverse problems with two composed INNs.

Original: Notably, [52] and [75] do not leverage alow-dimensional latent space, and [75] requires training a separate model for each image sample.
Revised: Notably, [52] and [77] do not leverage alow-dimensional latent space, and [77] requires training a separate model for each image sample.

Original: Inthis paper, we use INN to model arbitrary EBMs for various generative models.
Revised: Inthis paper, we use INN to model arbitrary EBMs for various generative models.

Original:  Method
Revised:  Method

Original: Figure 2 illustrates our PromptGen framework.
Revised: Figure 2 illustrates our PromptGen framework.

Original: To begin, the user selects a pre-trained generativemodel G .
Revised: To begin, the user selects a pre-trained generativemodel G .

Original: PromptGen then lets the user specify a control C as an energy-based model (EBM) p ( z |C ) .
Revised: PromptGen then lets the user specify a control C as an energy-based model (EBM) p ( z |C ) .

Original: We train an INN f ✓ to approximate p ( z |C ) .
Revised: We train an INN f ✓ to approximate p ( z |C ) .

Original: PromptGen views the functional composition G  f ✓ as anew generative model and can perform iterative control.
Revised: PromptGen views the functional composition G  f ✓ as anew generative model and can perform iterative control.

Original: Algorithm 1 describes the overall procedure.
Revised: Algorithm 1 describes the overall procedure.

Original: Algorithm
Revised: Algorithm

Original: Input: control C of the current iteration 2.
Revised: Input: control C of the current iteration 2.

Original: Deﬁne an EBM p ( z |C ) for C (Section 3.1) 3.
Revised: Deﬁne an EBM p ( z |C ) for C (Section 3.1) 3.

Original: Train an INN f ✓ : R d !
Revised: Train an INN f ✓ : R d !

Original: R d to approximate p ( z |C ) (Section 3.2) 4.
Revised: R d to approximate p ( z |C ) (Section 3.2) 4.

Original: G G  f ✓
Revised: G G  f ✓

Original: The plug-and-play generative model [47] was ﬁrst proposed to use a latent-space EBM for controllableimage synthesis.
Revised: The plug-and-play generative model [47] was ﬁrst proposed to use a latent-space EBM for controllableimage synthesis.

Original: If a ﬁxed generative model is used, then theoretically, any image-space EBM can beviewed as a latent-space EBM, as shown by [19] and [50].
Revised: If a ﬁxed generative model is used, then theoretically, any image-space EBM can beviewed as a latent-space EBM, as shown by [19] and [50].

Original: We deﬁne the latent-space EBM followingsimilar formulation as [47, 19, 50], with some new energy functions.
Revised: We deﬁne the latent-space EBM followingsimilar formulation as [47, 19, 50], with some new energy functions.

Original: We then extend the formulationto incorporate the moment constraint [7], which was adopted for language modeling [36], but unlike[36], we deﬁne the moment constraint in the latent space to accommodate generative vision models.
Revised: We then extend the formulationto incorporate the moment constraint [7], which was adopted for language modeling [36], but unlike[36], we deﬁne the moment constraint in the latent space to accommodate generative vision models.

Original: We deﬁne a control C as M independent properties { y 1 , . . .
Revised: We deﬁne a control C as M independent properties { y 1 , . . .

Original: , y M } , e.g., y 1 can be a text descriptionand y 2 can be an attribute.
Revised: , y M } , e.g., y 1 can be a text descriptionand y 2 can be an attribute.

Original: The controllability can be deﬁned as the EBM (detailed in Appendix B.1)
Revised: The controllability can be deﬁned as the EBM (detailed in Appendix B.1)

Original: [Equation]
Revised: [Equation]

Original: which reweights the image prior p x ( x ) with energy E C ( x ) , where images with smaller energy arepreferred.
Revised: which reweights the image prior p x ( x ) with energy E C ( x ) , where images with smaller energy arepreferred.

Original: Using a pre-trained generative model G : R d !
Revised: Using a pre-trained generative model G : R d !

Original: X that maps a latent code z to an imagex , the image prior p x ( x ) is deﬁned by sampling a latent code z from p z = N ( 0 , I ) and mapping itto x = G ( z ) .
Revised: X that maps a latent code z to an imagex , the image prior p x ( x ) is deﬁned by sampling a latent code z from p z = N ( 0 , I ) and mapping itto x = G ( z ) .

Original: Appendix B.2 shows that this EBM is equivalent to the latent-space EBMp
Revised: Appendix B.2 shows that this EBM is equivalent to the latent-space EBMp

Original: [Equation]
Revised: [Equation]

Original: |
Revised: |

Original: [Equation]
Revised: [Equation]

Original: Latent-space EBM allows us to use any off-the-shelf model to specify the control.
Revised: Latent-space EBM allows us to use any off-the-shelf model to specify the control.

Original: The following aresome examples that are discussed in this paper (explained in Appendix B.1):
Revised: The following aresome examples that are discussed in this paper (explained in Appendix B.1):

Original: Classiﬁer energy:
Revised: Classiﬁer energy:

Original: Given a
Revised: Given a

Original: classiﬁer
Revised: classiﬁer

Original: P ( ·| x ) and the target class a that we want to sample imagesfrom, we deﬁne the
Revised: P ( ·| x ) and the target class a that we want to sample imagesfrom, we deﬁne the

Original: classiﬁer energy
Revised: classiﬁer energy

Original: as E
Revised: as E

Original: classiﬁer
Revised: classiﬁer

Original: x
Revised: x

Original: , a ) =
Revised: , a ) =

Original:  log
Revised:  log

Original: P ( a | x ) .
Revised: P ( a | x ) .

Original: CLIP energy: Using the CLIP model [56], we deﬁne the CLIP energy as the cosine distance betweenthe embeddings of the image and the text t , averaged over L differentiable augmentations [81, 44]:
Revised: CLIP energy: Using the CLIP model [56], we deﬁne the CLIP energy as the cosine distance betweenthe embeddings of the image and the text t , averaged over L differentiable augmentations [83, 44]:

Original: [Equation]
Revised: [Equation]

Original: Inverse graphics energy:
Revised: Inverse graphics energy:

Original: Given an inverse graphics model, f X !
Revised: Given an inverse graphics model, f X !

Original: P , which infers image parameters 105 (e.g., pose and expression), and the target parameters ⇢ , we deﬁne the inverse graphics energy as
Revised: P , which infers image parameters 105 (e.g., pose and expression), and the target parameters ⇢ , we deﬁne the inverse graphics energy as

Original: [Equation]
Revised: [Equation]

Original: where d h · , · i is the geodesic distance between the inferred parameters and the target parameters.
Revised: where d h · , · i is the geodesic distance between the inferred parameters and the target parameters.

Original: Moment constraint: Some controls cannot be directly deﬁned by off-the-shelf models, and themoment constraint [7, 36] is one of them.
Revised: Moment constraint: Some controls cannot be directly deﬁned by off-the-shelf models, and themoment constraint [7, 36] is one of them.

Original: Given a mapping  : X !
Revised: Given a mapping  : X !

Original: R K (e.g.,  can be a classiﬁerthat outputs the probability simplex), the moment constraint deﬁnes the target distribution p ( x |C ) as
Revised: R K (e.g.,  can be a classiﬁerthat outputs the probability simplex), the moment constraint deﬁnes the target distribution p ( x |C ) as

Original: [Equation]
Revised: [Equation]

Original: where µ is the user-speciﬁed vector.
Revised: where µ is the user-speciﬁed vector.

Original: For example, if we want to generate images that are uniformlydistributed across races, we may use a race classiﬁer as  and deﬁne µ =  |A|  1 , . .
Revised: For example, if we want to generate images that are uniformlydistributed across races, we may use a race classiﬁer as  and deﬁne µ =  |A|  1 , . .

Original: , |A|  where
Revised: , |A|  where

Original: A is the set of races.
Revised: A is the set of races.

Original: In this paper, we generalize the moment constraint to the latent space, andapproximate the above objective as (detailed in Appendix B.6):
Revised: In this paper, we generalize the moment constraint to the latent space, andapproximate the above objective as (detailed in Appendix B.6):

Original: [Equation]
Revised: [Equation]

Original: 3.2 Approximating EBM with Invertible Neural Network
Revised: 3.2 Approximating EBM with Invertible Neural Network

Original: Given the functional form of the EBM p ( z |C ) , our next step is to approximate it with an efﬁcientsampling network.
Revised: Given the functional form of the EBM p ( z |C ) , our next step is to approximate it with an efﬁcientsampling network.

Original: To achieve this, we train a distribution p ✓ ( z ) that minimizes the KL divergence
Revised: To achieve this, we train a distribution p ✓ ( z ) that minimizes the KL divergence

Original: D KL ( p ✓ ( z ) k p ( z |C )) .
Revised: D KL ( p ✓ ( z ) k p ( z |C )) .

Original: Estimating D KL ( p ✓ ( z ) k p ( z |C )) requires easily sampling z ⇠ p ✓ and tractably 118 computing p ✓ ( z ) .
Revised: Estimating D KL ( p ✓ ( z ) k p ( z |C )) requires easily sampling z ⇠ p ✓ and tractably 118 computing p ✓ ( z ) .

Original: Inspired by [52], we model p ✓ with an INN f ✓ that deﬁnes a bijection z
Revised: Inspired by [52], we model p ✓ with an INN f ✓ that deﬁnes a bijection z

Original: = f ✓ ( ✏ ) ,which has two merits besides the invertibility: (1) one can easily sample z by sampling ✏ ⇠ N ( 0 , I )and mapping it to z = f ✓ ( ✏ ) , and (2) p ✓ ( z ) has a closed-form solution:
Revised: = f ✓ ( ✏ ) ,which has two merits besides the invertibility: (1) one can easily sample z by sampling ✏ ⇠ N ( 0 , I )and mapping it to z = f ✓ ( ✏ ) , and (2) p ✓ ( z ) has a closed-form solution:

Original: [Equation]
Revised: [Equation]

Original: [Figure]
Revised: [Figure]

Original: while not converged
Revised: while not converged

Original: Sample ✏ ⇠ N ( 0 , I ) 2.
Revised: Sample ✏ ⇠ N ( 0 , I ) 2.

Original: Map ✏ to latent code z = f ✓ ( ✏ ) 3.
Revised: Map ✏ to latent code z = f ✓ ( ✏ ) 3.

Original: Map z to an image x = G ( z ) 4.
Revised: Map z to an image x = G ( z ) 4.

Original: Optimize ✓ with gradient r ✓
Revised: Optimize ✓ with gradient r ✓

Original: [Equation]
Revised: [Equation]

Original: ⌘
Revised: ⌘

Original: Based on these properties of INN, we can rewrite our KL divergence objective D KL ( p ✓ ( z ) k p ( z |C ))as (full derivations in Appendix B.3)
Revised: Based on these properties of INN, we can rewrite our KL divergence objective D KL ( p ✓ ( z ) k p ( z |C ))as (full derivations in Appendix B.3)

Original: the following form:
Revised: the following form:

Original: [Equation]
Revised: [Equation]

Original: Since H N ( 0 , I ) and log
Revised: Since H N ( 0 , I ) and log

Original: Z are independent of ✓ , our training objective becomes
Revised: Z are independent of ✓ , our training objective becomes

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Figure 3 gives an illustration of our process, and Algorithm 2 describes the algorithmic details.
Revised: Figure 3 gives an illustration of our process, and Algorithm 2 describes the algorithmic details.

Original: PromptGen for generative models with a class-embedding space (Figure 5(e))
Revised: PromptGen for generative models with a class-embedding space (Figure 5(e))

Original: Previous works[4, 62] have shown that class conditioning boosts generative models’ performances on diverse datasuch as ImageNet [60].
Revised: Previous works[4, 62] have shown that class conditioning boosts generative models’ performances on diverse datasuch as ImageNet [60].

Original: Speciﬁcally, class-conditioned generative models map a latent code z and aclass embedding y to x = G ( z , y ) .
Revised: Speciﬁcally, class-conditioned generative models map a latent code z and aclass embedding y to x = G ( z , y ) .

Original: To extend PromptGen to these models, we train an INN h ✓ tomap ⇠ ⇠ N ( µ ,  2 I ) to y =
Revised: To extend PromptGen to these models, we train an INN h ✓ tomap ⇠ ⇠ N ( µ ,  2 I ) to y =

Original: h ✓ ( ⇠ ) , where µ and  are the mean and standard deviation of G ’s classembeddings.
Revised: h ✓ ( ⇠ ) , where µ and  are the mean and standard deviation of G ’s classembeddings.

Original: The motivation for deﬁning the distribution of ⇠ as N ( µ ,  2 I ) but not N ( 0 , I ) is thatwe want the learned INN h ✓ to be volume-preserving, which is easier to train.
Revised: The motivation for deﬁning the distribution of ⇠ as N ( µ ,  2 I ) but not N ( 0 , I ) is thatwe want the learned INN h ✓ to be volume-preserving, which is easier to train.

Original: The training objectiveis to minimize D KL ( p ✓ ( z , y ) k p
Revised: The training objectiveis to minimize D KL ( p ✓ ( z , y ) k p

Original: ( z , y |C )) , which is equivalent to (full derivations in Appendix B.4)
Revised: ( z , y |C )) , which is equivalent to (full derivations in Appendix B.4)

Original: [Equation]
Revised: [Equation]

Original: where p y ( y ) =
Revised: where p y ( y ) =

Original: N ( y | µ ,  2 I ) is the estimated class-embedding distribution.
Revised: N ( y | µ ,  2 I ) is the estimated class-embedding distribution.

Original: PromptGen with conditional INN To generalize the control to continuous values, e.g., the sceneparameters ⇢ in Eq.
Revised: PromptGen with conditional INN To generalize the control to continuous values, e.g., the sceneparameters ⇢ in Eq.

Original: (4), we condition the INN on ⇢ .
Revised: (4), we condition the INN on ⇢ .

Original: The condition is modeled by replacing z = f ✓ ( ✏ )with z = f ✓ ( ✏ , ⇢ ) .
Revised: The condition is modeled by replacing z = f ✓ ( ✏ )with z = f ✓ ( ✏ , ⇢ ) .

Original: Space limited, we provide details and derivations in Appendix B.5.
Revised: Space limited, we provide details and derivations in Appendix B.5.

Original: This extensionresults in a similar architecture to StyleFlow [1], but StyleFlow [1] uses MLE training on image-label
Revised: This extensionresults in a similar architecture to StyleFlow [1], but StyleFlow [1] uses MLE training on image-label

Original: [Figure]
Revised: [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original:  Experiments
Revised:  Experiments

Original: This section describes the experimental validation of PromptGen.
Revised: This section describes the experimental validation of PromptGen.

Original: See Appendix A for experimentson synthetic data and Appendix C, D, E and F for additional experiments on images and 3D meshes.
Revised: See Appendix A for experimentson synthetic data and Appendix C, D, E and F for additional experiments on images and 3D meshes.

Original: Additional experimental details are provided in Appendix B.8.
Revised: Additional experimental details are provided in Appendix B.8.

Original: 4.1 Image Synthesis based on Text Description
Revised: 4.1 Image Synthesis based on Text Description

Original: This experiment illustrates the capability of PromptGen to sample images from a generative modeldriven by a text description t using the pre-trained CLIP model [56] (the ViT-B/32 version).
Revised: This experiment illustrates the capability of PromptGen to sample images from a generative modeldriven by a text description t using the pre-trained CLIP model [56] (the ViT-B/32 version).

Original: We usedthe CLIP energy from Eq.
Revised: We usedthe CLIP energy from Eq.

Original: (3), with text descriptions such as a photo of a baby.
Revised: (3), with text descriptions such as a photo of a baby.

Original: Figure 4 shows a comparison between PromptGen and two previous CLIP-guided image generationmethods, StyleCLIP [54] and StyleGAN2-NADA [17].
Revised: Figure 4 shows a comparison between PromptGen and two previous CLIP-guided image generationmethods, StyleCLIP [54] and StyleGAN2-NADA [17].

Original: We observe that PromptGen generates diverseand high-quality images of babies, while StyleCLIP struggles in controllability and image quality, and
Revised: We observe that PromptGen generates diverseand high-quality images of babies, while StyleCLIP struggles in controllability and image quality, and

Original: StyleGAN2-NADA generates baby-like adults.
Revised: StyleGAN2-NADA generates baby-like adults.

Original: These results show that (1) locally editing the latentcode (i.e., StyleCLIP) is not always as effective method for controlling generative models (e.g., not allimages’ latent code can be locally edited into a baby); (2) domain adaptation (i.e., StyleGAN-NADA)is not effective in seeking modes in a generative model.
Revised: These results show that (1) locally editing the latentcode (i.e., StyleCLIP) is not always as effective method for controlling generative models (e.g., not allimages’ latent code can be locally edited into a baby); (2) domain adaptation (i.e., StyleGAN-NADA)is not effective in seeking modes in a generative model.

Original: Figure 5(a), Figure 5(b), and Figure 5(d)
Revised: Figure 5(a), Figure 5(b), and Figure 5(d)

Original: showhow PromptGen applies to StyleNeRF [21], NVAE [73], and diffusion autoencoder (DiffAE; [55]).
Revised: showhow PromptGen applies to StyleNeRF [21], NVAE [74], and diffusion autoencoder (DiffAE; [55]).

Original: Although these generative models are all trained on FFHQ, baby images sampled from them havedistinct characteristics.
Revised: Although these generative models are all trained on FFHQ, baby images sampled from them havedistinct characteristics.

Original: Figure 5(c) shows the ability of PromptGen to generate cats of a particularspecies.
Revised: Figure 5(c) shows the ability of PromptGen to generate cats of a particularspecies.

Original: Figure 5(e) shows the extension (Section 3.2) of our PromptGen to the class-embedding spaceof BigGAN
Revised: Figure 5(e) shows the extension (Section 3.2) of our PromptGen to the class-embedding spaceof BigGAN

Original: [4], a class-conditional GAN trained on ImageNet [60].
Revised: [4], a class-conditional GAN trained on ImageNet [60].

Original: We observe that PromptGenhelps BigGAN generate images with complex text descriptions, which are out of BigGAN’s trainingimage distribution; however, the diversity seem to be limited in this extension.
Revised: We observe that PromptGenhelps BigGAN generate images with complex text descriptions, which are out of BigGAN’s trainingimage distribution; however, the diversity seem to be limited in this extension.

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure]
Revised: [Figure]

Original: r o p o r t i o n
Revised: r o p o r t i o n

Original: Race
Revised: Race

Original: WhiteBlackLatinoHispanicEastAsian
Revised: WhiteBlackLatinoHispanicEastAsian

Original: SoutheastAsianIndianMiddleEastern
Revised: SoutheastAsianIndianMiddleEastern

Original: [Figure]
Revised: [Figure]

Original: t
Revised: t

Original: [Figure]
Revised: [Figure]

Original: n
Revised: n

Original: Race
Revised: Race

Original: WhiteBlackLatinoHispanicEastAsian
Revised: WhiteBlackLatinoHispanicEastAsian

Original: SoutheastAsianIndianMiddleEastern
Revised: SoutheastAsianIndianMiddleEastern

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: StyleGAN2 [35]
Revised: StyleGAN2 [35]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: StyleFlow [1]
Revised: StyleFlow [1]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: FairGen [70]
Revised: FairGen [70]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: FairStyle [30]
Revised: FairStyle [30]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: PromptGen
Revised: PromptGen

Original: [Table]
Revised: [Table]

Original: ours)
Revised: ours)

Original: [Table]
Revised: [Table]

Original: 4.2 De-Biasing Pre-Trained Generative Models
Revised: 4.2 De-Biasing Pre-Trained Generative Models

Original: An important problem in generative models is to generate fair distributions w.r.t a set of attributes ofinterest.
Revised: An important problem in generative models is to generate fair distributions w.r.t a set of attributes ofinterest.

Original: For instance, Figure 6(a) shows that StyleGAN2 generates images with bias across races andages.
Revised: For instance, Figure 6(a) shows that StyleGAN2 generates images with bias across races andages.

Original: PromptGen de-biases StyleGAN2 models trained on FFHQ 1024 2 and MetFaces 1024 2 [32] interms of categorical attributes, using the moment constraint deﬁned in Eq.
Revised: PromptGen de-biases StyleGAN2 models trained on FFHQ 1024 2 and MetFaces 1024 2 [32] interms of categorical attributes, using the moment constraint deﬁned in Eq.

Original: (5) and Eq.
Revised: (5) and Eq.

Original: As control,we used a classiﬁer trained on FairFace 224 2 [31] as  .
Revised: As control,we used a classiﬁer trained on FairFace 224 2 [31] as  .

Original: We deﬁned µ =  |A|  1 , . . .
Revised: We deﬁned µ =  |A|  1 , . . .

Original: , |A|  , where
Revised: , |A|  , where

Original: A is the set of races.
Revised: A is the set of races.

Original: Similar to the energy weights  i deﬁned in Eq.
Revised: Similar to the energy weights  i deﬁned in Eq.

Original: (2), we propose to rescale thetrained ˆ  as  ˆ  .
Revised: (2), we propose to rescale thetrained ˆ  as  ˆ  .

Original: Figure 6 shows that PromptGen de-biases the race and age effectively.
Revised: Figure 6 shows that PromptGen de-biases the race and age effectively.

Original: Existing de-biasing baselines consider binary attributes [30].
Revised: Existing de-biasing baselines consider binary attributes [30].

Original: For a fair comparison with them, weadopted their setting to use binary classiﬁers trained on CelebA [45] for de-biasing and evaluation.
Revised: For a fair comparison with them, weadopted their setting to use binary classiﬁers trained on CelebA [45] for de-biasing and evaluation.

Original: Since classiﬁers trained on CelebA also suffer from the spurious correlation between attributes, wedid not use the moment constraint for this experiment.
Revised: Since classiﬁers trained on CelebA also suffer from the spurious correlation between attributes, wedid not use the moment constraint for this experiment.

Original: Instead, since PromptGen allows conditionalimage generation with the classiﬁer energy, we generated the same number of samples conditioned oneach attribute or attribute combination.
Revised: Instead, since PromptGen allows conditionalimage generation with the classiﬁer energy, we generated the same number of samples conditioned oneach attribute or attribute combination.

Original: Table 2 shows that PromptGen has competitive performanceon de-biasing for attributes and attribute combinations.
Revised: Table 2 shows that PromptGen has competitive performanceon de-biasing for attributes and attribute combinations.

Original: 4.3 Pose-Guided Face Synthesis
Revised: 4.3 Pose-Guided Face Synthesis

Original: With an inverse graphics model, PromptGen can control the pose of faces generated by StyleGAN2.
Revised: With an inverse graphics model, PromptGen can control the pose of faces generated by StyleGAN2.

Original: We used the DECA model [16], which infers the parameters of FLAME [42], a parametric facialgraphics model.
Revised: We used the DECA model [16], which infers the parameters of FLAME [42], a parametric facialgraphics model.

Original: We set ⇢ 2 SO (3) as FLAME’s three neck poses and used the conditional INNextension introduced in Section 3.2.
Revised: We set ⇢ 2 SO (3) as FLAME’s three neck poses and used the conditional INNextension introduced in Section 3.2.

Original: To enable generating different poses of the same identity (ID),we propose an additional ID energy using the IR-SE50 model [8].
Revised: To enable generating different poses of the same identity (ID),we propose an additional ID energy using the IR-SE50 model [8].

Original: Speciﬁcally, given a canonicalpose ⇢ 0 , we deﬁne z 

Revised: Speciﬁcally, given a canonicalpose ⇢ 0 , we deﬁne z

Original: =
Revised: =

Original: f ✓ ( ✏ , ⇢ 0 ) , and the ID energy is deﬁned as (detailed in Appendix B.7)
Revised: f ✓ ( ✏ , ⇢ 0 ) , and the ID energy is deﬁned as (detailed in Appendix B.7)

Original: [Equation]
Revised: [Equation]

Original: where R is the IR-SE50 model [8] that computes face embeddings.
Revised: where R is the IR-SE50 model [8] that computes face embeddings.

Original: Figure 7 shows that PromptGengenerates faces of the same ID in different poses, even without being explicitly trained with poses ascondition.
Revised: Figure 7 shows that PromptGengenerates faces of the same ID in different poses, even without being explicitly trained with poses ascondition.

Original: We reported the FID score [23] of each model, computed with Clean-FID [53]; following controls the pose of StyleGAN preserving the identity
Revised: We reported the FID score [23] of each model, computed with Clean-FID [53]; following controls the pose of StyleGAN preserving the identity

Original: All images are resized for visualization
Revised: All images are resized for visualization

Original: [Table]
Revised: [Table]

Original: [35], we did not use the truncation trick when computing the FID score.
Revised: [35], we did not use the truncation trick when computing the FID score.

Original: Table 3 shows that PromptGenoutperforms existing models in terms of the FID score.
Revised: Table 3 shows that PromptGenoutperforms existing models in terms of the FID score.

Original: Following [67], we then reported the average
Revised: Following [67], we then reported the average

Original: IR-SE50 [8] embedding distances for images with the same ID and with different IDs.
Revised: IR-SE50 [8] embedding distances for images with the same ID and with different IDs.

Original: Results showthat PromptGen achieves the best ID preservation, with a slight sacriﬁce of ID diversity.
Revised: Results showthat PromptGen achieves the best ID preservation, with a slight sacriﬁce of ID diversity.

Original: 4.4 Decomposing Complex Control via Energy Composition
Revised: 4.4 Decomposing Complex Control via Energy Composition

Original: Energy composition E
Revised: Energy composition E

Original: [Equation]
Revised: [Equation]

Original: =
Revised: =

Original: P
Revised: P

Original: Mi
Revised: Mi

Original: [Equation]
Revised: [Equation]

Original: 
Revised:

Original: [Equation]
Revised: [Equation]

Original: E i ( x , y i ) allows us to decompose controls into simple ones.
Revised: E i ( x , y i ) allows us to decompose controls into simple ones.

Original: We show that y 

Revised: We show that y

Original: = photo of a bald black man with beard (  1 = 6000 ) is not successful (Figure 8(a));by decomposing it as y 1 = photo of a bald man (  1 = 1500 ), y 

Revised: = photo of a bald black man with beard (  1 = 6000 ) is not successful (Figure 8(a));by decomposing it as y 1 = photo of a bald man (  1 = 1500 ), y

Original: = photo of a black man (  1 =3000 ), and y 

Revised: = photo of a black man (  1 =3000 ), and y

Original: = photo of a man with beard (  1 = 1500 ), we have better control (Figure 8(b)).
Revised: = photo of a man with beard (  1 = 1500 ), we have better control (Figure 8(b)).

Original: [Figure]
Revised: [Figure]

Original: 4.5 Iterative Distributional Control via Functional Composition
Revised: 4.5 Iterative Distributional Control via Functional Composition

Original: This section discusses an interesting bias that PromptGen reveals about the CLIP model.
Revised: This section discusses an interesting bias that PromptGen reveals about the CLIP model.

Original: Figure 9(a)shows images generated by PromptGen (with StyleGAN2) with the CLIP model and the text description a photo of a person without makeup, where more females are generated than males.
Revised: Figure 9(a)shows images generated by PromptGen (with StyleGAN2) with the CLIP model and the text description a photo of a person without makeup, where more females are generated than males.

Original: Thisbias should not be attributed to image pre-training data since images contain a bias in the oppositedirection, i.e., men are less likely to have makeup.
Revised: Thisbias should not be attributed to image pre-training data since images contain a bias in the oppositedirection, i.e., men are less likely to have makeup.

Original: We argue that this bias should be explained by
Revised: We argue that this bias should be explained by

Original: CLIP having learned a “reporting bias” in vision-language pre-training data : people are more likelyto say “a person without makeup” when the person is a female (detailed analysis in Appendix F).
Revised: CLIP having learned a “reporting bias” in vision-language pre-training data : people are more likelyto say “a person without makeup” when the person is a female (detailed analysis in Appendix F).

Original: Besides revealing the above “reporting bias”, PromptGen can also mitigate this bias via an iterativecontrol, enabled by the functional composition in Algorithm 1.
Revised: Besides revealing the above “reporting bias”, PromptGen can also mitigate this bias via an iterativecontrol, enabled by the functional composition in Algorithm 1.

Original: Speciﬁcally, in the second iteration,we de-biased the gender distribution of G  f ✓ instead of G , where f ✓ is the INN learned for the textcontrol.
Revised: Speciﬁcally, in the second iteration,we de-biased the gender distribution of G  f ✓ instead of G , where f ✓ is the INN learned for the textcontrol.

Original: For de-biasing, we used the moment constraint with ˆ  trained for G  f ✓ .
Revised: For de-biasing, we used the moment constraint with ˆ  trained for G  f ✓ .

Original: Figure 9(b) showsthat females and males are uniformly distributed after the moment constraint in the second iteration.
Revised: Figure 9(b) showsthat females and males are uniformly distributed after the moment constraint in the second iteration.

Original: [Figure]
Revised: [Figure]

Original: PromptGen generates more images male, showing
Revised: PromptGen generates more images male, showing

Original: Besides revealing this “reporting bias”, PromptGen can also mitigate this bias via an iterative control, allowing us to de-bias the text-controlled distribution.
Revised: Besides revealing this “reporting bias”, PromptGen can also mitigate this bias via an iterative control, allowing us to de-bias the text-controlled distribution.

Original: All images are resized for visualization.
Revised: All images are resized for visualization.

Original: 4.6 Inference Latency
Revised: 4.6 Inference Latency

Original: We compared our PromptGen and the plug-and-play generative model (PPGM) [47] in terms of theinference latency.
Revised: We compared our PromptGen and the plug-and-play generative model (PPGM) [47] in terms of theinference latency.

Original: PPGM uses Langevin dynamics [74] to optimize over the latent space at inference,while PromptGen samples images in a feed-forward manner.
Revised: PPGM uses Langevin dynamics [75] to optimize over the latent space at inference,while PromptGen samples images in a feed-forward manner.

Original: In this experiment, we used PPGM andour PromptGen to approximate the same EBM with CLIP energy.
Revised: In this experiment, we used PPGM andour PromptGen to approximate the same EBM with CLIP energy.

Original: Inference times was estimated onan NVIDIA RTX A4000 GPU.
Revised: Inference times was estimated onan NVIDIA RTX A4000 GPU.

Original: Table 4 shows that our PromptGen has the highest performance andefﬁciency.
Revised: Table 4 shows that our PromptGen has the highest performance andefﬁciency.

Original: Compared to our PromptGen, PPGM requires 100 ⇥ inference time to achieve comparableresults.
Revised: Compared to our PromptGen, PPGM requires 100 ⇥ inference time to achieve comparableresults.

Original: Moreover, learning f ✓ in PromptGen requires training-time optimization, and this amortizedoptimization is useful when one wants to reuse a controlled distribution many times.
Revised: Moreover, learning f ✓ in PromptGen requires training-time optimization, and this amortizedoptimization is useful when one wants to reuse a controlled distribution many times.

Original: [Table]
Revised: [Table]

Original: [Table]
Revised: [Table]

Original: When is it equal to the EBM?
Revised: When is it equal to the EBM?

Original: [Equation]
Revised: [Equation]

Original:  Conclusions and Future Work
Revised:  Conclusions and Future Work

Original: This paper proposes PromptGen, a uniﬁed framework to learn latent distributions for distributionalcontrol of pre-trained generative models.
Revised: This paper proposes PromptGen, a uniﬁed framework to learn latent distributions for distributionalcontrol of pre-trained generative models.

Original: PromptGen leverages the knowledge of various off-the-shelfmodels and, unlike previous methods, it does not require the availability of these models at inference.
Revised: PromptGen leverages the knowledge of various off-the-shelfmodels and, unlike previous methods, it does not require the availability of these models at inference.

Original: PromptGen can sample images in a feed-forward manner, which is more efﬁcient than methodsthat require optimization at inference.
Revised: PromptGen can sample images in a feed-forward manner, which is more efﬁcient than methodsthat require optimization at inference.

Original: PromptGen offers the generality for algorithmic design andmodularity for control composition, and it also enables iterative controls.
Revised: PromptGen offers the generality for algorithmic design andmodularity for control composition, and it also enables iterative controls.

Original: Experiments validate that
Revised: Experiments validate that

Original: PromptGen applies to various generative models (StyleGAN2, StyleNeRF, NVAE), control types(continuous, discrete, and moment constraint), off-the-shelf models (CLIP, classiﬁers, and inversegraphics models), and data domains (faces, churches, animals, ImageNet, and landscapes).
Revised: PromptGen applies to various generative models (StyleGAN2, StyleNeRF, NVAE), control types(continuous, discrete, and moment constraint), off-the-shelf models (CLIP, classiﬁers, and inversegraphics models), and data domains (faces, churches, animals, ImageNet, and landscapes).

Original: Limitation and future work
Revised: Limitation and future work:

Original: We provide an error analysis in Appendix F. PromptGen is restrictedby the pre-trained generative model’s coverage [3, 27].
Revised: We provide an error analysis in Appendix F and a discussion onsocietal impact in Appendix G. PromptGen is restricted by the pre-trained generative model’s coverage[3, 27].

Original: PromptGen focuses on mode-seeking and-reweighting instead of domain adaptation, and it is possible to combine PromptGen and domainadaptation of generative models [17].
Revised: PromptGen focuses on mode-seeking and -reweighting instead of domain adaptation, and it ispossible to combine PromptGen and domain adaptation of generative models [17].

Original: Also, PromptGen depends on the off-the-shelf models thatprovide knowledge about the control (Appendix F).
Revised: Also, PromptGendepends on the off-the-shelf models that provide knowledge about the control (Appendix F).

Original: In subsequent work, it can be beneﬁcial to explorehow to learn energy functions [15, 26] (besides our moment constraint used for de-biasing), whichmay provide ﬁne-grained control with less bias.
Revised: Insubsequent work, it can be beneﬁcial to explore how to learn energy functions [15, 26] (besides ourmoment constraint used for de-biasing), which may provide ﬁne-grained control with less bias.