Original: Prompt Generation: Uniﬁed Distributional Control of Pre-Trained GenerativeVision Models
Revised: Generative Visual Prompt: Unifying Distributional Control of Pre-Trained Generative Models

Original: Anonymous Author(s)
Revised: Anonymous Author(s)

Original: AfﬁliationAddress email
Revised: AfﬁliationAddress email

Original: Abstract
Revised: Abstract

Original: Generative models (e.g., GANs, VAEs) learn the underlying data distribution inan unsupervised manner.
Revised: Generative models (e.g., GANs, VAEs) learn the underlying data distribution inan unsupervised manner.

Original: However, many applications of interest require samplingfrom a speciﬁc region of the generative model’s output space or evenly over arange of characteristics.
Revised: However, many applications of interest require samplingfrom a speciﬁc region of the generative model’s output space or evenly over arange of characteristics.

Original: To allow efﬁcient sampling in these scenarios, we propose
Revised: To allow efﬁcient sampling in these scenarios, we propose

Original: Prompt Generation (PromptGen), a framework to achieve distributional controlover pre-trained generative models by incorporating knowledge of arbitrary offthe-shelf models.
Revised: Prompt Generation (PromptGen), a framework to achieve distributional controlover pre-trained generative models by incorporating knowledge of arbitrary offthe-shelf models.

Original: PromptGen deﬁnes control as an energy-based model (EBM)and samples images in a feed-forward manner by approximating the EBM withinvertible neural networks, avoiding optimization at inference.
Revised: PromptGen deﬁnes control as an energy-based model (EBM)and samples images in a feed-forward manner by approximating the EBM withinvertible neural networks, avoiding optimization at inference.

Original: We demonstratehow PromptGen can control several generative models using various off-the-shelfmodels: (1) with the CLIP model, PromptGen can sample images guided by text,(2) with image classiﬁers, PromptGen can de-bias generative models across a set ofattributes, and (3) with inverse graphics models, PromptGen can sample images ofthe same identity in different poses.
Revised: We demonstratehow PromptGen can control several generative models using various off-the-shelfmodels: (1) with the CLIP model, PromptGen can sample images guided by text,(2) with image classiﬁers, PromptGen can de-bias generative models across a set ofattributes, and (3) with inverse graphics models, PromptGen can sample images ofthe same identity in different poses.

Original: (4) Finally, PromptGen reveals that the CLIPmodel shows “reporting bias” when used as control, and PromptGen can furtherde-bias this controlled distribution in an iterative manner.
Revised: (4) Finally, PromptGen reveals that the CLIPmodel shows “reporting bias” when used as control, and PromptGen can furtherde-bias this controlled distribution in an iterative manner.

Original:  Introduction
Revised:  Introduction

Original: Generative models (e.g., GANs, VAEs) learn the underlying data distribution in high-dimensionalspaces and achieve promising performance on image synthesis [4, 71, 35, 21].
Revised: Generative models (e.g., GANs, VAEs) learn the underlying data distribution in high-dimensionalspaces and achieve promising performance on image synthesis [4, 73, 35, 21].

Original: Though being wellpraised, they still face two main criticisms.
Revised: Though being wellpraised, they still face two main criticisms.

Original: First, since generative models are typically trained in anunsupervised way, they lack controllability, meaning that it is unclear how to sample from a speciﬁcregion of the space.
Revised: First, since generative models are typically trained in anunsupervised way, they lack controllability, meaning that it is unclear how to sample from a speciﬁcregion of the space.

Original: Second, generative models are prone to inherit the imbalance and bias of trainingdata [57, 30].
Revised: Second, generative models are prone to inherit the imbalance and bias of trainingdata [58, 30].

Original: For instance, StyleGAN2 is more likely to produce images of white individuals, see
Revised: For instance, StyleGAN2 is more likely to produce images of white individuals, see

Original: Figure 1(e).
Revised: Figure 1(e).

Original: Previous works have studied these challenges separately, and typical methods includeediting of “style” codes [22, 64, 30] and explicit conditions [41, 65].
Revised: Previous works have studied these challenges separately, and typical methods includeediting of “style” codes [22, 66, 30] and explicit conditions [41, 67].

Original: However, these methods areeither model-dependent (i.e., requiring a well-structured style space) or label-intensive (i.e., requiringall training samples to be labeled for explicit conditions), limiting their generality and practical use.
Revised: However, these methods areeither model-dependent (i.e., requiring a well-structured style space) or label-intensive (i.e., requiringall training samples to be labeled for explicit conditions), limiting their generality and practical use.

Original: To address the above challenges, this paper advocates a uniﬁed formulation, distributional controlof generative models , which enables controllability by incorporating the knowledge of off-the-shelfmodels (e.g., CLIP [55], classiﬁers, or inverse graphics models [16]).
Revised: To address the above challenges, this paper advocates a uniﬁed formulation, distributional controlof generative models , which enables controllability by incorporating the knowledge of off-the-shelfmodels (e.g., CLIP [56], classiﬁers, or inverse graphics models [16]).

Original: Based on this uniﬁed view, wepropose to learn distributions in the latent space of a pre-trained generative model while keeping thepre-trained parameters ﬁxed.
Revised: Based on this uniﬁed view, wepropose to learn distributions in the latent space of a pre-trained generative model while keeping thepre-trained parameters ﬁxed.

Original: Given its conceptual similarity to prompt learning [40, 79, 70, 80, 29],we term our framework as Prompt Generation (PromptGen).
Revised: Given its conceptual similarity to prompt learning [40, 82, 72, 83, 29],we term our framework as Prompt Generation (PromptGen).

Original: PromptGen requires no training data ,and the only supervision comes from off-the-shelf models that help deﬁne the control.
Revised: PromptGen requires no training data ,and the only supervision comes from off-the-shelf models that help deﬁne the control.

Original: Speciﬁcally,
Revised: Speciﬁcally,

Original: PromptGen allows the user to deﬁne controls using an energy-based model (EBM) approximatedSubmitted to by an invertible neural network (INN).
Revised: PromptGen allows the user to deﬁne controls using an energy-based model (EBM) approximated by an invertible neural network (INN).

Original: Unlike methods that require optimization at inference in
Revised: Unlike methods that require optimization at inference in

Original: EBM sampling [47, 14, 43, 50], PromptGen samples images in a feed-forward manner, which ishighly efﬁcient.
Revised: EBM sampling [47, 14, 43, 50], PromptGen samples images in a feed-forward manner, which ishighly efﬁcient.

Original: Moreover, PromptGen stands alone at inference, meaning that we can discard theoff-the-shelf models, which deﬁne the control, after training.
Revised: Moreover, PromptGen stands alone at inference, meaning that we can discard theoff-the-shelf models, which deﬁne the control, after training.

Original: We illustrate the beneﬁts of PromptGen with several experiments.
Revised: We illustrate the beneﬁts of PromptGen with several experiments.

Original: Figure 1 demonstrates our mainﬁndings with StyleGAN2 trained on FFHQ [34] without any labels, while we also show results for
Revised: Figure 1 demonstrates our mainﬁndings with StyleGAN2 trained on FFHQ [34] without any labels, while we also show results for

Original: StyleNeRF
Revised: StyleNeRF

Original: [21] and NVAE [71] on non-facial data in Section 4.
Revised: [21] and NVAE [73] on non-facial data in Section 4.

Original: Figure 1(a) illustrates how wecan sample from StyleGAN2 based on text descriptions such as “a photo of a baby”.
Revised: Figure 1(a) illustrates how wecan sample from StyleGAN2 based on text descriptions such as “a photo of a baby”.

Original: Figure 1(b)shows that PromptGen can leverage a race classiﬁer (potentially trained on a different dataset) tosample uniformly across all races, de-biasing the pre-trained StyleGAN2.
Revised: Figure 1(b)shows that PromptGen can leverage a race classiﬁer (potentially trained on a different dataset) tosample uniformly across all races, de-biasing the pre-trained StyleGAN2.

Original: Figure 1(d) illustrates that
Revised: Figure 1(d) illustrates that

Original: PromptGen can generate images of the same identity in different poses, guided by a pose regressor.
Revised: PromptGen can generate images of the same identity in different poses, guided by a pose regressor.

Original: Finally, it is worth pointing out that PromptGen not only offers generality for algorithmic designand modularity for control composition, but also enables iterative controls when some controls arecontingent on others.
Revised: Finally, it is worth pointing out that PromptGen not only offers generality for algorithmic designand modularity for control composition, but also enables iterative controls when some controls arecontingent on others.

Original: For instance, one may train a text-controlled distribution and then de-bias thisdistribution.
Revised: For instance, one may train a text-controlled distribution and then de-bias thisdistribution.

Original: To achieve this, we view the composition of the INN and the generative model as a new“generative model” to be controlled.
Revised: To achieve this, we view the composition of the INN and the generative model as a new“generative model” to be controlled.

Original: Figure 1(f) illustrates that PromptGen reveals “reporting bias” ofthe CLIP model [55], where “without makeup” is – perhaps surprisingly – positively correlated withfemale, and Figure 1(c) shows that PromptGen can further mitigate this bias with iterative control.
Revised: Figure 1(f) illustrates that PromptGen reveals “reporting bias” ofthe CLIP model [56], where “without makeup” is – perhaps surprisingly – positively correlated withfemale, and Figure 1(c) shows that PromptGen can further mitigate this bias with iterative control.

Original: [Table]
Revised: [Table]

Original:  Related Work
Revised:  Related Work

Original: Over the past few years, generative models have gained the ability to generate images with high visualquality.
Revised: Over the past few years, generative models have gained the ability to generate images with high visualquality.

Original: A few of the most widely used methods include generative adversarial networks (GANs) [18],
Revised: A few of the most widely used methods include generative adversarial networks (GANs) [18],

Original: VAE [38], invertible neural networks [11, 12], and diffusion models [24, 67].
Revised: VAE [38], invertible neural networks [11, 12], and diffusion models [24, 69].

Original: In particular, generativemodels trained on large amounts of unlabeled data, e.g., BigGAN [4], StyleGANs [34, 35, 33, 61],
Revised: In particular, generativemodels trained on large amounts of unlabeled data, e.g., BigGAN [4], StyleGANs [34, 35, 33, 62],

Original: Glow [37], and diffusion models [49, 10], achieve promising image synthesis results.
Revised: Glow [37], and diffusion models [49, 10], achieve promising image synthesis results.

Original: Despite their success, controllability and de-biasing are still two fundamental challenges generativemodels face.
Revised: Despite their success, controllability and de-biasing are still two fundamental challenges generativemodels face.

Original: For controllability, existing methods include explicit conditioning at training [41, 65] or local editing of the learned representation, e.g, “style” codes [1, 54, 44, 64, 74].
Revised: For controllability, existing methods include explicit conditioning at training [41, 67] or local editing of the learned representation, e.g, “style” codes [1, 54, 44, 66, 77].

Original: For de-biasing,existing methods include local editing of “style” codes [57, 30] and importance sampling for eithertraining [20] or inference [28].
Revised: For de-biasing,existing methods include local editing of “style” codes [58, 30] and importance sampling for eithertraining [20] or inference [28].

Original: Existing works study these problems separately, each requiring aspeciﬁc design for the task studied.
Revised: Existing works study these problems separately, each requiring aspeciﬁc design for the task studied.

Original: On the contrary, PromptGen is a uniﬁed framework for arbitrarycontrols deﬁned by off-the-shelf models.
Revised: On the contrary, PromptGen is a uniﬁed framework for arbitrarycontrols deﬁned by off-the-shelf models.

Original: The beneﬁts of unifyingrelated tasks have been shown bya recent trend of works in multiple research areas across vision and language [56, 46, 60, 75, 58].
Revised: The beneﬁts of unifying tasks have been shown by a recenttrend of works in multiple research areas across vision and language [57, 46, 61, 78, 59].

Original: Moreover, one can ﬁne-tunepre-trained generative models [17] to achieve domain adaptation, whichis orthogonal to PromptGen: since PromptGen outputs a generative model  (Algorithm 1), ﬁne-tuningcan be applied afterwards.
Revised: Moreover,one can ﬁne-tune generative models [17] for domain adaptation, which is orthogonal to PromptGen:since PromptGen maps a generative model to another generative model (Algorithm 1), ﬁne-tuningcan be applied before or after PromptGen training.

Original: However, we leave this exploration to future studies.
Revised: We leave this exploration to future studies.

Original: Previous methods usually sample from EBM [39] with Markov Chain Monte Carlo (MCMC) [69, 72,14, 19, 13, 43].
Revised: Previous methods usually sample from EBM [39] with Markov Chain Monte Carlo (MCMC) [71, 74,14, 19, 13, 43].

Original: Among them, plug-and-play generative models (PPGMs) [47] and LACE [50] deﬁnelatent-space EBMs.
Revised: Among them, plug-and-play generative models (PPGMs) [47] and LACE [50] deﬁnelatent-space EBMs.

Original: However, MCMC requires inference-time optimization, which is inefﬁcient andrequires the off-the-shelf models to be available at inference; this is also the criticism for diffusionmodels, regardless of being gradient-guided [49, 63] or not [24, 25, 48].
Revised: However, MCMC requires inference-time optimization, which is inefﬁcient andrequires the off-the-shelf models to be available at inference; this is also the criticism for diffusionmodels, regardless of being gradient-guided [49, 65] or not [24, 25, 48].

Original: In contrast, PromptGenachieves efﬁcient, feed-forward sampling.
Revised: In contrast, PromptGenachieves efﬁcient, feed-forward sampling.

Original: Table 1 shows a comparison with previous methods.
Revised: Table 1 shows a comparison with previous methods.

Original: Our
Revised: Our

Original: INN training is similar to that proposed by [52] to sample from physical systems, which is later usedby [73] to solve inverse problems with two composed INNs.
Revised: INN training is similar to that proposed by [52] to sample from physical systems, which is later usedby [75] to solve inverse problems with two composed INNs.

Original: Notably, [52] and [73] do not leverage alow-dimensional latent space, and [73] requires training a separate model for each image sample.
Revised: Notably, [52] and [75] do not leverage alow-dimensional latent space, and [75] requires training a separate model for each image sample.

Original: Inthis paper, we use INN to model arbitrary EBMs for various generative models.
Revised: Inthis paper, we use INN to model arbitrary EBMs for various generative models.

Original:  Method
Revised:  Method

Original: Figure 2 illustrates our PromptGen framework.
Revised: Figure 2 illustrates our PromptGen framework.

Original: To begin, the user selects a pre-trained generativemodel G .
Revised: To begin, the user selects a pre-trained generativemodel G .

Original: PromptGen then lets the user specify a control C as an energy-based model (EBM) p ( z |C ) .
Revised: PromptGen then lets the user specify a control C as an energy-based model (EBM) p ( z |C ) .

Original: We train an INN f ✓ to approximate p ( z |C ) .
Revised: We train an INN f ✓ to approximate p ( z |C ) .

Original: PromptGen views the functional composition G  f ✓ as anew generative model and can perform iterative control.
Revised: PromptGen views the functional composition G  f ✓ as anew generative model and can perform iterative control.

Original: Algorithm 1 describes the overall procedure. 
Revised: Algorithm 1 describes the overall procedure. Algorithm

Original: Input: control C of the current iteration 2.
Revised: Input: control C of the current iteration 2.

Original: Deﬁne an EBM p ( z |C ) for C (Section 3.1) 3.
Revised: Deﬁne an EBM p ( z |C ) for C (Section 3.1) 3.

Original: Train an INN f ✓ : R d !
Revised: Train an INN f ✓ : R d !

Original: R d to approximate p ( z |C ) (Section 3.2) 4.
Revised: R d to approximate p ( z |C ) (Section 3.2) 4.

Original: G G  f ✓
Revised: G G  f ✓

Original: The plug-and-play generative model [47] was ﬁrst proposed to use a latent-space EBM for controllableimage synthesis.
Revised: The plug-and-play generative model [47] was ﬁrst proposed to use a latent-space EBM for controllableimage synthesis.

Original: If a ﬁxed generative model is used, then theoretically, any image-space EBM can beviewed as a latent-space EBM, as shown by [19] and [50].
Revised: If a ﬁxed generative model is used, then theoretically, any image-space EBM can beviewed as a latent-space EBM, as shown by [19] and [50].

Original: We deﬁne the latent-space EBM followingsimilar formulation as [47, 19, 50], with some new energy functions.
Revised: We deﬁne the latent-space EBM followingsimilar formulation as [47, 19, 50], with some new energy functions.

Original: We then extend the formulationto incorporate the moment constraint [36], which was adopted for language modeling, but unlike[36], we deﬁne the moment constraint in the latent space to accommodate generative vision models.
Revised: We then extend the formulationto incorporate the moment constraint [7], which was adopted for language modeling [36], but unlike[36], we deﬁne the moment constraint in the latent space to accommodate generative vision models.

Original: We deﬁne a control C as M independent properties { y 1 , . . .
Revised: We deﬁne a control C as M independent properties { y 1 , . . .

Original: , y M } , e.g., y 1 can be a text descriptionand y 2 can be an attribute.
Revised: , y M } , e.g., y 1 can be a text descriptionand y 2 can be an attribute.

Original: The controllability can be deﬁned as the EBM (detailed in Appendix B.1)
Revised: The controllability can be deﬁned as the EBM (detailed in Appendix B.1)

Original: [Equation]
Revised: [Equation]

Original: which reweights the image prior p x ( x ) with energy E C ( x ) , where images with smaller energy arepreferred.
Revised: which reweights the image prior p x ( x ) with energy E C ( x ) , where images with smaller energy arepreferred.

Original: Using a pre-trained generative model G : R d !
Revised: Using a pre-trained generative model G : R d !

Original: X that maps a latent code z to an imagex , the image prior p x ( x ) is deﬁned by sampling a latent code z from p z = N ( 0 , I ) and mapping itto x = G ( z ) .
Revised: X that maps a latent code z to an imagex , the image prior p x ( x ) is deﬁned by sampling a latent code z from p z = N ( 0 , I ) and mapping itto x = G ( z ) .

Original: Appendix B.2 shows that this EBM is equivalent to the latent-space EBMp
Revised: Appendix B.2 shows that this EBM is equivalent to the latent-space EBMp

Original: [Equation]
Revised: [Equation]

Original: |
Revised: |

Original: [Equation]
Revised: [Equation]

Original: Latent-space EBM allows us to use any off-the-shelf model to specify the control.
Revised: Latent-space EBM allows us to use any off-the-shelf model to specify the control.

Original: The following aresome examples that are discussed in this paper (explained in Appendix B.1):
Revised: The following aresome examples that are discussed in this paper (explained in Appendix B.1):

Original: Classiﬁer energy:
Revised: Classiﬁer energy:

Revised: Given a

Original: Given a classiﬁer P ( ·| x ) and the target class a that we want to sample imagesfrom, we deﬁne the classiﬁer energy as E
Revised: classiﬁer P ( ·| x ) and the target class a that we want to sample imagesfrom, we deﬁne the classiﬁer energy as E

Original: classiﬁer
Revised: classiﬁer

Original: ( x , a ) =
Revised: x , a ) =

Original:  log P ( a | x ) .
Revised:  log P ( a | x ) .

Original: CLIP energy: Using the CLIP model [55], we deﬁne the CLIP energy as the cosine distance betweenthe embeddings of the image and the text y , averaged over L differentiable augmentations [78, 44]:
Revised: CLIP energy: Using the CLIP model [56], we deﬁne the CLIP energy as the cosine distance betweenthe embeddings of the image and the text t , averaged over L differentiable augmentations [81, 44]:

Original: [Equation]
Revised: [Equation]

Original: Inverse graphics energy:
Revised: Inverse graphics energy:

Original: Given an inverse graphics model, f X !
Revised: Given an inverse graphics model, f X !

Original: P , which infers image parameters 105 (e.g., pose and expression), and the target parameters ⇢ , we deﬁne the inverse graphics energy as
Revised: P , which infers image parameters 105 (e.g., pose and expression), and the target parameters ⇢ , we deﬁne the inverse graphics energy as

Original: [Equation]
Revised: [Equation]

Original: where d h · , · i is the geodesic distance between the inferred parameters and the target parameters.
Revised: where d h · , · i is the geodesic distance between the inferred parameters and the target parameters.

Original: Moment constraint: Some controls cannot be directly deﬁned by off-the-shelf models, and themoment constraint [7, 36] is one of them.
Revised: Moment constraint: Some controls cannot be directly deﬁned by off-the-shelf models, and themoment constraint [7, 36] is one of them.

Original: Given a mapping  : X !
Revised: Given a mapping  : X !

Original: R K (e.g.,  can be a classiﬁerthat outputs the probability simplex), the moment constraint deﬁnes the target distribution p ( x |C ) as
Revised: R K (e.g.,  can be a classiﬁerthat outputs the probability simplex), the moment constraint deﬁnes the target distribution p ( x |C ) as

Original: [Equation]
Revised: [Equation]

Original: where µ is the user-speciﬁed vector.
Revised: where µ is the user-speciﬁed vector.

Original: For example, if we want to generate images that are uniformlydistributed across races, we may use a race classiﬁer as  and deﬁne µ =  |A|  1 , . .
Revised: For example, if we want to generate images that are uniformlydistributed across races, we may use a race classiﬁer as  and deﬁne µ =  |A|  1 , . .

Original: , |A|  where
Revised: , |A|  where

Original: A is the set of races.
Revised: A is the set of races.

Original: In this paper, we generalize the moment constraint to the latent space, andapproximate the above objective as (detailed in Appendix B.6):
Revised: In this paper, we generalize the moment constraint to the latent space, andapproximate the above objective as (detailed in Appendix B.6):

Original: [Equation]
Revised: [Equation]

Original: 3.2 Approximating EBM with Invertible Neural Network
Revised: 3.2 Approximating EBM with Invertible Neural Network

Original: Given the functional form of the EBM p ( z |C ) , our next step is to approximate it with an efﬁcientsampling network.
Revised: Given the functional form of the EBM p ( z |C ) , our next step is to approximate it with an efﬁcientsampling network.

Original: To achieve this, we train a distribution p ✓ ( z ) that minimizes the KL divergence
Revised: To achieve this, we train a distribution p ✓ ( z ) that minimizes the KL divergence

Original: D KL ( p ✓ ( z ) k p ( z |C )) .
Revised: D KL ( p ✓ ( z ) k p ( z |C )) .

Original: Estimating D KL ( p ✓ ( z ) k p ( z |C )) requires easily sampling z ⇠ p ✓ and tractably 118 computing p ✓ ( z ) .
Revised: Estimating D KL ( p ✓ ( z ) k p ( z |C )) requires easily sampling z ⇠ p ✓ and tractably 118 computing p ✓ ( z ) .

Original: Inspired by [52], we model p ✓ with an INN f ✓ that deﬁnes a bijection z
Revised: Inspired by [52], we model p ✓ with an INN f ✓ that deﬁnes a bijection z

Original: = f ✓ ( ✏ ) ,which has two merits besides the invertibility: (1) one can easily sample z by sampling ✏ ⇠ N ( 0 , I )and mapping it to z = f ✓ ( ✏ ) , and (2) p ✓ ( z ) has a closed-form solution:
Revised: = f ✓ ( ✏ ) ,which has two merits besides the invertibility: (1) one can easily sample z by sampling ✏ ⇠ N ( 0 , I )and mapping it to z = f ✓ ( ✏ ) , and (2) p ✓ ( z ) has a closed-form solution:

Original: [Equation]
Revised: [Equation]

Original: (cid:44)(cid:49)(cid:49)(cid:3) (cid:3)(cid:11)(cid:82)(cid:83)(cid:87)(cid:76)(cid:82)(cid:81)(cid:68)(cid:79)(cid:12) while not convergeddo
Revised: [Figure] while not converged

Original: Sample ✏ ⇠ N ( 0 , I ) 2.
Revised: Sample ✏ ⇠ N ( 0 , I ) 2.

Original: Map ✏ to latent code z = f ✓ ( ✏ ) 3.
Revised: Map ✏ to latent code z = f ✓ ( ✏ ) 3.

Original: Map z to an image x = G ( z ) 4.
Revised: Map z to an image x = G ( z ) 4.

Original: Optimize ✓ with gradient r ✓
Revised: Optimize ✓ with gradient r ✓

Original: [Equation]
Revised: [Equation]

Original: ⌘
Revised: ⌘

Original: Based on these properties of INN, we can rewrite our KL divergence objective D KL ( p ✓ ( z ) k p ( z |C ))as (full derivations in Appendix B.3)
Revised: Based on these properties of INN, we can rewrite our KL divergence objective D KL ( p ✓ ( z ) k p ( z |C ))as (full derivations in Appendix B.3)

Original: the following form:
Revised: the following form:

Original: [Equation]
Revised: [Equation]

Original: Since H N ( 0 , I ) and log
Revised: Since H N ( 0 , I ) and log

Original: Z are independent of ✓ , our training objective becomesarg
Revised: Z are independent of ✓ , our training objective becomes

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Original: Figure 3 gives an illustration of our process, and Algorithm 2 describes the algorithmic details.
Revised: Figure 3 gives an illustration of our process, and Algorithm 2 describes the algorithmic details.

Original: PromptGen for generative models with a class-embedding space (Figure 5(d))
Revised: PromptGen for generative models with a class-embedding space (Figure 5(e))

Original: Previous works[4, 61] have shown that class conditioning boosts generative models’ performances on diverse datasuch as ImageNet [59].
Revised: Previous works[4, 62] have shown that class conditioning boosts generative models’ performances on diverse datasuch as ImageNet [60].

Original: Speciﬁcally, class-conditioned generative models map a latent code z and aclass embedding y to x = G ( z , y ) .
Revised: Speciﬁcally, class-conditioned generative models map a latent code z and aclass embedding y to x = G ( z , y ) .

Original: To extend PromptGen to these models, we train an INN h ✓ tomap ⇠ ⇠ N ( µ ,  2 I ) to y =
Revised: To extend PromptGen to these models, we train an INN h ✓ tomap ⇠ ⇠ N ( µ ,  2 I ) to y =

Original: h ✓ ( ⇠ ) , where µ and  are the mean and standard deviation of G ’s classembeddings.
Revised: h ✓ ( ⇠ ) , where µ and  are the mean and standard deviation of G ’s classembeddings.

Original: The motivation for deﬁning the distribution of ⇠ as N ( µ ,  2 I ) but not N ( 0 , I ) is thatwe want the learned INN h ✓ to be volume-preserving, which is easier to train.
Revised: The motivation for deﬁning the distribution of ⇠ as N ( µ ,  2 I ) but not N ( 0 , I ) is thatwe want the learned INN h ✓ to be volume-preserving, which is easier to train.

Original: The training objectiveis to minimize D KL ( p ✓ ( z , y ) k p
Revised: The training objectiveis to minimize D KL ( p ✓ ( z , y ) k p

Original: ( z , y |C )) , which is equivalent to (full derivations in Appendix B.4)
Revised: ( z , y |C )) , which is equivalent to (full derivations in Appendix B.4)

Original: [Equation]
Revised: [Equation]

Original: where p y ( y ) =
Revised: where p y ( y ) =

Original: N ( y | µ ,  2 I ) is the estimated class-embedding distribution.
Revised: N ( y | µ ,  2 I ) is the estimated class-embedding distribution.

Original: PromptGen with conditional INN When the control is speciﬁed by continuous values, e.g., thescene parameters ⇢ in Eq.
Revised: PromptGen with conditional INN To generalize the control to continuous values, e.g., the sceneparameters ⇢ in Eq.

Original: (4), it is desirable to condition the INN on ⇢ because it helps in generalizingto all continuous values.
Revised: (4), we condition the INN on ⇢ .

Original: Speciﬁcally, the conditioning is implemented bysimply replacing z = f ✓ ( ✏ )with z = f ✓ ( ✏ , ⇢ ) .
Revised: The condition is modeled by replacing z = f ✓ ( ✏ )with z = f ✓ ( ✏ , ⇢ ) .

Original: Full derivations are Appendix B. PromptGen with conditional INNresults architecture
Revised: Space limited, we provide details and derivations in Appendix B.5.

Original:  StyleFlow [1] StyleFlow [1]  training image-labelpairs and is only applicable to explicit conditioning. Therefore, StyleFlow is not capable of modeling
Revised: This extensionresults in a similar architecture to StyleFlow [1], but StyleFlow [1] uses MLE training on image-label

Original: EBMs, which are required for CLIP control and moment constraint.

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: Figure 5(c) is PromptGen applied to StyleGAN2 [35] trained on AFHQ-Cats [6]. Figure d is the extension to BigGAN [4] trained on ImageNet [59] C for on datasets
Revised: [Figure]

Original:  Experiments
Revised:  Experiments

Original: This section describes the experimental validation of PromptGen.
Revised: This section describes the experimental validation of PromptGen.

Original: See Appendix A for experimentson synthetic data and Appendix C, D, E, and F for additional experiments on images and 3D meshes.
Revised: See Appendix A for experimentson synthetic data and Appendix C, D, E and F for additional experiments on images and 3D meshes.

Original: Additional experimental details are provided in Appendix B.8.
Revised: Additional experimental details are provided in Appendix B.8.

Original: 4.1 Image Synthesis based on Text Description
Revised: 4.1 Image Synthesis based on Text Description

Original: This experiment illustrates the capability of PromptGen to sample images from a generative modeldriven by a text description y using the pre-trained CLIP model [55].
Revised: This experiment illustrates the capability of PromptGen to sample images from a generative modeldriven by a text description t using the pre-trained CLIP model [56] (the ViT-B/32 version).

Original: We used the CLIP energy from Eq.
Revised: We usedthe CLIP energy from Eq.

Original: (3), with text descriptions such as a photo of a baby.
Revised: (3), with text descriptions such as a photo of a baby.

Original: Figure 4 shows a comparison between PromptGen and two previous CLIP-guided image generationmethods, StyleCLIP [54] and StyleGAN2-NADA [17].
Revised: Figure 4 shows a comparison between PromptGen and two previous CLIP-guided image generationmethods, StyleCLIP [54] and StyleGAN2-NADA [17].

Original: All methods used StyleGAN2 as the backbone.

Original: We observe that PromptGen generates diverse and high-quality images of babies, while StyleCLIPstruggles in controllability and image quality, and StyleGAN2-NADA generates baby-like adults.
Revised: We observe that PromptGen generates diverseand high-quality images of babies, while StyleCLIP struggles in controllability and image quality, and StyleGAN2-NADA generates baby-like adults.

Original: Figure 5(a) and Figure 5(b) display how our PromptGen framework applies to StyleNeRF [21], a NeRF-based GAN model, and NVAE [71], a deep variational autoencoder model.
Revised: These results show that (1) locally editing the latentcode (i.e., StyleCLIP) is not always as effective method for controlling generative models (e.g., not allimages’ latent code can be locally edited into a baby); (2) domain adaptation (i.e., StyleGAN-NADA)is not effective in seeking modes in a generative model.

Original: We observe that
Revised: Figure 5(a), Figure 5(b), and Figure 5(d)

Original:  PromptGen with StyleNeRF (Figure 5(a)) has higher diversity than PromptGen with StyleGAN2(Figure 4(c)), and PromptGen with NVAE has lower image quality.
Revised: showhow PromptGen applies to StyleNeRF [21], NVAE [73], and diffusion autoencoder (DiffAE; [55]). Although these generative models are all trained on FFHQ, baby images sampled from them havedistinct characteristics.

Original: Figure 5(c) shows the abilityof PromptGen to generate cat faces based on the text description.
Revised: Figure 5(c) shows the ability of PromptGen to generate cats of a particularspecies.

Original: Figure 5(d) shows theresults forthe extension (Section 3.2) of our PromptGenframework to  BigGAN [4], a class-conditional GANtrained on ImageNet [59] with a class-embedding space.
Revised: Figure 5(e) shows the extension (Section 3.2) of our PromptGen to the class-embedding spaceof BigGAN [4], a class-conditional GAN trained on ImageNet [60].

Original: Text descriptions used for this experimentare shown below each set of images. We observe that PromptGen helps BigGAN generate imageswith complex text descriptions, which are out of BigGAN’s pre-training image distribution.
Revised: We observe that PromptGenhelps BigGAN generate images with complex text descriptions, which are out of BigGAN’s trainingimage distribution; however, the diversity seem to be limited in this extension.

Original: [Figure]
Revised: [Figure]

Original: [Figure][Figure] [Figure] [Figure]
Revised: [Figure]

Original: [Figure]
Revised: [Figure]

Original: r o p o r t i o n
Revised: r o p o r t i o n

Original: Race
Revised: Race

Original: WhiteBlackLatinoHispanicEastAsian
Revised: WhiteBlackLatinoHispanicEastAsian

Original: SoutheastAsianIndianMiddleEastern
Revised: SoutheastAsianIndianMiddleEastern

Original: [Figure]
Revised: [Figure]

Original: n
Revised: t

Original: [Figure]
Revised: [Figure]

Original: t i o n
Revised: n

Original: Race
Revised: Race

Original: WhiteBlackLatinoHispanicEastAsian
Revised: WhiteBlackLatinoHispanicEastAsian

Original: SoutheastAsianIndianMiddleEastern
Revised: SoutheastAsianIndianMiddleEastern

Original: [Figure]
Revised: [Figure]

Revised: [Figure]

Revised: [Table]

Revised: [Table]

Revised: StyleGAN2 [35]

Revised: [Table]

Revised: [Table]

Revised: StyleFlow [1]

Revised: [Table]

Revised: [Table]

Revised: FairGen [70]

Revised: [Table]

Revised: [Table]

Revised: FairStyle [30]

Revised: [Table]

Revised: [Table]

Revised: PromptGen

Revised: [Table]

Revised: ours)

Original: [Table]
Revised: [Table]

Original: 4.2 De-biasing Generative Models
Revised: 4.2 De-Biasing Pre-Trained Generative Models

Original: An important problem in generative models is to generate fair distributions w.r.t a set of attributes ofinterest.
Revised: An important problem in generative models is to generate fair distributions w.r.t a set of attributes ofinterest.

Original: For instance, Figure 6(a) shows that StyleGAN2 generates images with bias across races andages.
Revised: For instance, Figure 6(a) shows that StyleGAN2 generates images with bias across races andages.

Original: PromptGen de-biases StyleGAN2 models trained on FFHQ 1024 2 and MetFaces 1024 2 [32] interms of categorical attributes, using the moment constraint deﬁned in Eq.
Revised: PromptGen de-biases StyleGAN2 models trained on FFHQ 1024 2 and MetFaces 1024 2 [32] interms of categorical attributes, using the moment constraint deﬁned in Eq.

Original: (5) and Eq.
Revised: (5) and Eq.

Original: As control,we used a classiﬁer trained on FairFace 224 2 [31] as  .
Revised: As control,we used a classiﬁer trained on FairFace 224 2 [31] as  .

Original: We deﬁned µ =  |A|  1 , . . .
Revised: We deﬁned µ =  |A|  1 , . . .

Original: , |A|  , where
Revised: , |A|  , where

Original: A is the set of races.
Revised: A is the set of races.

Original: Similar to the energy weights  i deﬁned in Eq.
Revised: Similar to the energy weights  i deﬁned in Eq.

Original: (2), we propose to rescale thetrained ˆ  as  ˆ  .
Revised: (2), we propose to rescale thetrained ˆ  as  ˆ  .

Original: Figure 6visualizes the attribute distribution before and after PromptGen de-biasing,which shows that PromptGen de-biases the race and agedistributions effectively.
Revised: Figure 6 shows that PromptGen de-biases the race and age effectively.

Original: Existing de-biasing baselines consider binary attributes [30].
Revised: Existing de-biasing baselines consider binary attributes [30].

Original: For a fair comparison with them, weadopted their setting to use binary classiﬁers trained on CelebA [45] for de-biasing and evaluation.
Revised: For a fair comparison with them, weadopted their setting to use binary classiﬁers trained on CelebA [45] for de-biasing and evaluation.

Original: Since classiﬁers trained on CelebA also suffer from the spurious correlation between attributes, wedid not use the moment constraint for this experiment.
Revised: Since classiﬁers trained on CelebA also suffer from the spurious correlation between attributes, wedid not use the moment constraint for this experiment.

Original: Instead, since PromptGen allows conditionalimage generation with the classiﬁer energy, we generated the same number of samples conditioned oneach attribute or attribute combination.
Revised: Instead, since PromptGen allows conditionalimage generation with the classiﬁer energy, we generated the same number of samples conditioned oneach attribute or attribute combination.

Original: Table 2 shows that PromptGen has competitive performanceon de-biasing for attributes and attribute combinations.
Revised: Table 2 shows that PromptGen has competitive performanceon de-biasing for attributes and attribute combinations.

Original: In cases where the previous best FairStyle hasnearly perfect performance, the margin between PromptGen and FairStyle is only a few samples.

Original: 4.3 Pose-Guided Face Synthesis
Revised: 4.3 Pose-Guided Face Synthesis

Original: With an inverse graphics model, PromptGen can control the pose of faces generated by StyleGAN2trained on FFHQ 1024 2 .
Revised: With an inverse graphics model, PromptGen can control the pose of faces generated by StyleGAN2.

Original: Speciﬁcally, we used the DECA model [16], which infers the parameters of FLAME [42], a parametric facial graphics model with six pose parameters.
Revised: We used the DECA model [16], which infers the parameters of FLAME [42], a parametric facialgraphics model.

Original: We set ⇢ 2 SO (3) as FLAME’s three neck poses(but we kept the z -axis rotation ﬁxed since FFHQ does not contain suchvariation) and used the conditional INN extension introduced in Section 3.2.
Revised: We set ⇢ 2 SO (3) as FLAME’s three neck poses and used the conditional INNextension introduced in Section 3.2.

Original: To enable generatingdifferent poses of the same identity (ID), we propose an additional ID energy using the IR-SE50model [8].
Revised: To enable generating different poses of the same identity (ID),we propose an additional ID energy using the IR-SE50 model [8].

Original: Speciﬁcally, given a canonical pose ⇢ 0 , we deﬁne z 

Revised: Speciﬁcally, given a canonicalpose ⇢ 0 , we deﬁne z

Original: =
Revised: =

Original: f ✓ ( ✏ , ⇢ 0 ) , and the ID energy isdeﬁned as (detailed in Appendix B.7)
Revised: f ✓ ( ✏ , ⇢ 0 ) , and the ID energy is deﬁned as (detailed in Appendix B.7)

Original: [Equation]
Revised: [Equation]

Original: [Figure] [Table] where R is the IR-SE50 model [8] that computes face embeddings.
Revised: where R is the IR-SE50 model [8] that computes face embeddings.

Original: Figure 7 shows that PromptGengenerates faces of the same ID in different poses, even without being explicitly trained with poses ascondition.
Revised: Figure 7 shows that PromptGengenerates faces of the same ID in different poses, even without being explicitly trained with poses ascondition.

Original: We reported the FID score [23] of each model, computed with Clean-FID [53]; following[35], we did not use the truncation trick when computing the FID score.
Revised: We reported the FID score [23] of each model, computed with Clean-FID [53]; following controls the pose of StyleGAN preserving the identity All images are resized for visualization [Table] [35], we did not use the truncation trick when computing the FID score.

Original: Table 3 shows that PromptGenoutperforms existing models in terms of the FID score.
Revised: Table 3 shows that PromptGenoutperforms existing models in terms of the FID score.

Original: Following [65], we then reported the average
Revised: Following [67], we then reported the average

Original: IR-SE50 [8] embedding distances for images with the same ID and with different IDs.
Revised: IR-SE50 [8] embedding distances for images with the same ID and with different IDs.

Original: Results showthat PromptGen achieves the best ID preservation, with a slight sacriﬁce of ID diversity.
Revised: Results showthat PromptGen achieves the best ID preservation, with a slight sacriﬁce of ID diversity.

Original: 4.4 Iterative Distributional Control via Functional Composition
Revised: 4.4 Decomposing Complex Control via Energy Composition

Revised: Energy composition E [Equation] =

Revised: P

Revised: Mi

Revised: [Equation]

Original: This section discusses an interesting bias that our PromptGen reveals about the CLIP model.{+y 
+}
Revised:  [Equation] E i ( x , y i ) allows us to decompose controls into simple ones. We show that {+y 
+}

Original: Figure 8(a) shows images generated by PromptGen (with StyleGAN2) using the CLIP model with the{+beard (  1 = 6000 ) is not successful (Figure 8(a));by decomposing it as y 1 = photo of a bald man (  1 = 1500 ), y 
 = photo of a black man (  1 =3000 ), and y 
 = photo of a man with beard (  1 = 1500 ), we have better control (Figure 8(b)).+}
Revised: = photo of a bald black man with {+beard (  1 = 6000 ) is not successful (Figure 8(a));by decomposing it as y 1 = photo of a bald man (  1 = 1500 ), y 
 = photo of a black man (  1 =3000 ), and y 
 = photo of a man with beard (  1 = 1500 ), we have better control (Figure 8(b)).+}

Original: [Figure]
Revised: [Figure]

Original:  text description a photo of a person without makeup, where more females are generated than males.
Revised: This section discusses an interesting bias that PromptGen reveals about the CLIP model. Figure 9(a)shows images generated by PromptGen (with StyleGAN2) with the CLIP model and the text description a photo of a person without makeup, where more females are generated than males.

Original: This bias should not be attributed to image pre-training data because images contain a bias in the opposite direction, i.e., men are less likely to have makeup.
Revised: Thisbias should not be attributed to image pre-training data since images contain a bias in the oppositedirection, i.e., men are less likely to have makeup.

Original: We argue that this bias should be explainedby CLIP having learned a “reporting bias” in the multi-modal (vision-language) pre-training data :people are more likely to say “a person without makeup” when the person is a female.
Revised: We argue that this bias should be explained by CLIP having learned a “reporting bias” in vision-language pre-training data : people are more likelyto say “a person without makeup” when the person is a female (detailed analysis in Appendix F).

Original: Besides revealing the above “reporting bias”, PromptGen can also mitigate this bias via an iterativedistributional control, enabled by the functional composition in Algorithm 1.
Revised: Besides revealing the above “reporting bias”, PromptGen can also mitigate this bias via an iterativecontrol, enabled by the functional composition in Algorithm 1.

Original: Speciﬁcally, in thesecond iteration, we de-biased the gender distribution of G  f ✓ instead of G , where f ✓ is the INNlearned for the text control.
Revised: Speciﬁcally, in the second iteration,we de-biased the gender distribution of G  f ✓ instead of G , where f ✓ is the INN learned for the textcontrol.

Original: For de-biasing, we used the moment constraint, where ˆ is trained basedon G  f ✓ .
Revised: For de-biasing, we used the moment constraint with ˆ  trained for G  f ✓ .

Original: Figure 8(b) shows that females and males are uniformly distributed after the de-biasingcontrol in the second iteration, and all generated images do not have makeup, as originally requested.
Revised: Figure 9(b) showsthat females and males are uniformly distributed after the moment constraint in the second iteration. [Figure] PromptGen generates more images male, showing

Revised: Besides revealing this “reporting bias”, PromptGen can also mitigate this bias via an iterative control, allowing us to de-bias the text-controlled distribution.

Revised: All images are resized for visualization.

Original: 4.5 Inference Latency
Revised: 4.6 Inference Latency

Original: We compared our PromptGen and the plug-and-play generative model (PPGM) [47] in terms of theinference latency.
Revised: We compared our PromptGen and the plug-and-play generative model (PPGM) [47] in terms of theinference latency.

Original: A difference between the two methods is that PPGM uses Langevin dynamics [72]to optimize over the latent space during inference, while PromptGen samples images in a feed-forwardmanner.
Revised: PPGM uses Langevin dynamics [74] to optimize over the latent space at inference,while PromptGen samples images in a feed-forward manner.

Original: In this experiment, we used PPGM and our PromptGen to approximate the same EBM with CLIP energy.
Revised: In this experiment, we used PPGM andour PromptGen to approximate the same EBM with CLIP energy.

Original: Inference timesin Table 4 was estimated on an NVIDIA RTX A4000 GPU.
Revised: Inference times was estimated onan NVIDIA RTX A4000 GPU.

Original: Table 4shows that our PromptGen has the highest performance and efﬁciency.
Revised: Table 4 shows that our PromptGen has the highest performance andefﬁciency.

Original: Compared to our PromptGen, PPGM requires 100 ⇥ inference time to achieve comparable results.
Revised: Compared to our PromptGen, PPGM requires 100 ⇥ inference time to achieve comparableresults.

Original: was estimated on an NVIDIA RTX A4000 GPU.

Original: PPGM ( n = 10 ) PPGM ( n [Table] PromptGen 
Revised: Moreover, learning f ✓ in PromptGen requires training-time optimization, and this amortizedoptimization is useful when one wants to reuse a controlled distribution many times.

Original: [Table]

Original: ours)

Original: [Table]

Original: Back

Original: [Table]
Revised: [Table]

Original: through CLIP

Original: [Table]
Revised: [Table]

Original: [-inference  
-]
Revised: [-inference  
-]

Original: When is it equal to the EBM?
Revised: When is it equal to the EBM?

Original: n ! n ! D KL ( p ✓ ( z ) k p ( z |C )) =
Revised: [Equation]

Original:  Conclusions and Future Work
Revised:  Conclusions and Future Work

Original: This paper proposes PromptGen, a uniﬁed framework to learn latent distributions for distributionalcontrol of pre-trained generative models.
Revised: This paper proposes PromptGen, a uniﬁed framework to learn latent distributions for distributionalcontrol of pre-trained generative models.

Original: PromptGen leverages the knowledge of various off-the-shelfmodels and, unlike previous methods, it does not require the availability of these models at inference.
Revised: PromptGen leverages the knowledge of various off-the-shelfmodels and, unlike previous methods, it does not require the availability of these models at inference.

Original: PromptGen can sample images in a feed-forward manner, which is more efﬁcient than methodsthat require optimization at inference.
Revised: PromptGen can sample images in a feed-forward manner, which is more efﬁcient than methodsthat require optimization at inference.

Original: PromptGen offers the generality for algorithmic design andmodularity for control composition, and it also enables iterative controls.
Revised: PromptGen offers the generality for algorithmic design andmodularity for control composition, and it also enables iterative controls.

Original: Experiments validate that
Revised: Experiments validate that

Original: PromptGen applies to various generative models (StyleGAN2, StyleNeRF, NVAE), control types(continuous, discrete, and moment constraint), off-the-shelf models (CLIP, classiﬁers, and inversegraphics models), and data domains (faces, churches, animals, ImageNet, and landscapes).
Revised: PromptGen applies to various generative models (StyleGAN2, StyleNeRF, NVAE), control types(continuous, discrete, and moment constraint), off-the-shelf models (CLIP, classiﬁers, and inversegraphics models), and data domains (faces, churches, animals, ImageNet, and landscapes).

Original: Limitation and future work
Revised: Limitation and future work

Original: We provide an error analysis in Appendix F.Since PromptGenbuildsupon pre-trained generative models, it is inevitably restricted by the original generative model’scoverage [3, 27].
Revised: We provide an error analysis in Appendix F. PromptGen is restrictedby the pre-trained generative model’s coverage [3, 27].

Original: Except for the case of generative models with a class-embedding space, PromptGenfocuses on mode-seeking and mode-reweighting instead of domain adaptation. One possibledirectionis to ﬁne-tune the pre-trained generative model [17] after PromptGen learning to achieve domainadaptation.
Revised: PromptGen focuses on mode-seeking and-reweighting instead of domain adaptation, and it is possible to combine PromptGen and domainadaptation of generative models [17].

Original: Also, the control performance depends on thenature of off-the-shelf models that provideknowledge about the control. PromptGen helps us understand these off-the-shelf models, e.g., the CLIP model spuriously correlates female with the mention of “without makeup” (Figure 8(a)), andwe also show that it has difﬁculty in modeling linguistic negation (Appendix F).
Revised: Also, PromptGen depends on the off-the-shelf models thatprovide knowledge about the control (Appendix F).

Original: In subsequent work,it can be beneﬁcial to explore learnable energy functions [15, 26] (besides our moment constraint forde-biasing), which may provide ﬁne-grained control with less bias.
Revised: In subsequent work, it can be beneﬁcial to explorehow to learn energy functions [15, 26] (besides our moment constraint used for de-biasing), whichmay provide ﬁne-grained control with less bias.